Anomaly detection using a data stream processing language for analyzing instrumented software

ABSTRACT

An instrumentation analysis system processes data streams by executing instructions specified using a data stream language program. The data stream language allows users to specify a search condition using a find block for identifying the set of data streams processed by the data stream language program. The set of identified data streams may change dynamically. The data stream language allows users to group data streams into sets of data streams based on distinct values of one or more metadata attributes associated with the input data streams. The data stream language allows users to specify a threshold block for determining whether data values of input data streams are outside boundaries specified using low/high thresholds. The elements of the set of data streams input to the threshold block can dynamically change. The low/high threshold values can be specified as data streams and can dynamically change.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 62/094,935 filed Dec. 19, 2015, which is incorporated byreference in its entirety.

BACKGROUND

This disclosure relates to a data stream processing in general and morespecifically to a data stream processing language for processing datastreams received from instrumented software.

Software developers monitor different aspects of software they developby instrumenting the software. These include performance of thesoftware, errors encountered during execution of the software,significant events encountered during execution of the software,information describing which parts of code are being executed and whichparts are not being executed, and so on. Conventional techniques forinstrumenting code include statements in the code that log differenttypes of information to log files or print information on screens. Thistechnique is suitable for simple applications, for example, applicationshaving a simple flow of execution that execute on a single processor.However, these techniques for instrumenting software are inadequate forcomplex applications that may be distributed across multiple systems,each system executing multiple processes or threads of execution.

Another conventional technique for instrumenting such complex systems isto use help of experts in instrumenting code. Certain vendors provideexpert services that help with instrumentation of code. However, thesevendors typically provide standard services that are often not veryflexible. Furthermore, these vendor based solutions have significantoverhead in terms of time needed by the vendor to instrument code.Accordingly, these solutions are suited towards a slow developmentcycle, for example, a year-long development cycle. However, softwaredevelopment and release cycles for software products have become short.For example, there are several online systems in which softwaredevelopers make changes on a monthly, weekly, or even daily basis anddeploy them. Due to the significant overhead of vendor basedinstrumentation solutions, developers find it difficult to use theseservices in a fast paced development environment.

Furthermore, conventional techniques for instrumenting code causesignificant delays in assimilating the information, storing theinformation, and analyzing the information to generate reports. As aresult, there can be significant delay between the time that a problemoccurs in the software and the time that the problem is detected viainstrumentation of the code. Accordingly, conventional systems forgenerating reports based on instrumentation of software are ofteninadequate in fast paced development cycles of complex applications.

SUMMARY

Embodiments of an instrumentation analysis system process data streamsbased on instructions specified in a data stream language. The datastreams are received from instrumented code executing on externalsystems or may be generated by the instrumented analysis system asresults of data stream language programs. The instrumentation analysissystem receives a data stream language program that performs comparisonof data streams with threshold values. For example, data of data streamsmay be compared against a low threshold and/or a high threshold value.The low/high threshold values may be constant values or dynamicallychanging values. The low/high threshold values may be specified by adata stream language program that generates data streams.

The threshold block is configured to receive a first input and a secondinput. For example, the first input is received by a data port of thethreshold block and the second input is received by a threshold port ofthe threshold block. A first set of instructions of the data streamlanguage program generates the first input and a second set ofinstructions of the data stream language program generates the secondinput. The system receives an input set of data streams. The systemexecutes the first set of instructions to aggregate data of the inputset of data streams to generate a first plurality of data streamscomprising data values provided as the first input of the thresholdblock. The system executes the second set of instructions to aggregatedata of the input set of data streams to generate a second plurality ofdata streams comprising threshold values provided as the second input ofthe threshold block. The system matches data streams received as thefirst input as data values with data streams received at the secondinput as threshold values. For each data stream received at the firstinput, the system compares a data value of the data stream against athreshold value from a corresponding data stream from the secondplurality and determines whether to generate an event based on theresult of comparison of the data value and the threshold value.

In an embodiment, the event generated is reported as an anomaly detectedin the data streams analyzed by the system. For example, an anomaly mayindicate that a particular data stream or an aggregate value based on aset of data streams exceeded bounds set by low and high thresholds of athreshold block. The low and high thresholds may themselves changedynamically, for example, the low and high thresholds may be defined asmoving averages based on certain input data streams.

The features and advantages described in the specification are not allinclusive and in particular, many additional features and advantageswill be apparent to one of ordinary skill in the art in view of thedrawings, specification, and claims. Moreover, it should be noted thatthe language used in the specification has been principally selected forreadability and instructional purposes, and may not have been selectedto delineate or circumscribe the disclosed subject matter.

BRIEF DESCRIPTION OF DRAWINGS

The disclosed embodiments have other advantages and features which willbe more readily apparent from the detailed description, the appendedclaims, and the accompanying figures (or drawings). A brief introductionof the figures is below.

FIG. 1 shows the overall system environment for reporting based oninstrumented software, according to an embodiment.

FIG. 2 shows the architecture of a system for executing a data streamlanguage program for processing data streams received from instrumentedsoftware, according to an embodiment.

FIG. 3 shows the architecture the data stream language processor forprocessing blocks of data stream language programs, according to anembodiment.

FIG. 4 shows an example of a data stream language program forillustrating features of the data stream language, according to anembodiment.

FIG. 5 shows the overall process of an instrumentation analysis systemfor processing data received from data streams based on a data streamlanguage program, according to an embodiment.

FIG. 6 illustrates the process of quantization of the data streamsreceived from instrumented software, according to an embodiment.

FIG. 7 illustrates selection of a set of data streams by a find blockfor providing input to a data stream language program, according to anembodiment.

FIG. 8 illustrates dynamic changes to the set of data streams providinginput to a data stream language program as a result of periodicre-evaluation of the find block, according to an embodiment.

FIG. 9 shows the process for identifying a set of data streams forproviding input to a data stream language program using the find block,according to an embodiment.

FIG. 10 illustrates the process of retrieving data from data streams byexecuting a fetch block, according to an embodiment.

FIGS. 11A-C illustrate the process of combining data from the timeseries data store and data received in real-time from data streams formoving window calculations, according to an embodiment.

FIG. 12 illustrates a process for grouping data of data streams togenerate a set of result data streams, according to an embodiment.

FIGS. 13A-B shows an example scenario illustrating grouping of datastreams based on different metadata attributes describing the datastreams, according to an embodiment.

FIG. 14 shows an example scenario illustrating dynamic changing ofresult data streams generated by a group by block as a result of changesin input data streams over time, according to an embodiment.

FIG. 15 shows a flowchart illustrating the process of publishing resultdata streams obtained by executing a publish block of a data streamlanguage program, according to an embodiment.

FIG. 16 shows an example of a data stream language program illustratinguse of a threshold block with fixed threshold values for data streamsgrouped by a particular attribute, according to an embodiment.

FIG. 17 shows an example of a data stream language program illustratinguse of a threshold block with dynamically changing threshold values fordata streams grouped by metadata attributes, according to an embodiment.

FIG. 18 shows a flowchart illustrating the process of executing a datastream language program including a threshold block, according to anembodiment.

FIG. 19 shows an example of a data stream language program illustratinguse of a customized block for generating a result data stream based on auser defined function applied to inputs comprising groups of datastreams, according to an embodiment.

FIG. 20 shows a flowchart illustrating the process of executing a datastream language program with a customized block, according to anembodiment.

FIG. 21 shows a screenshot of a user interface displaying result ofexecution of a data stream language program that shows data streamsreceived by the instrumentation analysis system, according to anembodiment.

FIG. 22 shows a screenshot of a user interface displaying result ofexecution of a data stream language program showing 1 minute average ofdata of data streams received by the instrumentation analysis system,according to an embodiment.

FIG. 23 shows a screenshot of a user interface displaying result ofexecution of a data stream language program showing sum of data streamsgrouped by data center, according to an embodiment.

FIG. 24 shows a screenshot of a user interface displaying result ofexecution of a data stream language program including a customized macroblock that determines ratio of cache hit rate and sum of cache hit rateand miss rate for data streams grouped by datacenters, according to anembodiment.

Reference will now be made in detail to several embodiments, examples ofwhich are illustrated in the accompanying figures. It is noted thatwherever practicable similar or like reference numbers may be used inthe figures and may indicate similar or like functionality. The figuresdepict embodiments of the disclosed system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

DETAILED DESCRIPTION

Overall System Environment

FIG. 1 shows the overall system environment for reporting based oninstrumented software, according to an embodiment. The overall systemenvironment includes an instrumentation analysis system 100, one or moredevelopment systems 120, an administration system 160, and a reportingsystem 150. In other embodiments, more or less components than thoseindicated in FIG. 1 may be used. For example, development system 120,administration system 160, and reporting system 150 may interact withinstrumentation analysis system 100 via a network (not shown in FIG. 1).Furthermore, there may be more or less instances of each system shown inFIG. 1, for example, there may be multiple reporting systems 150.

FIG. 1 and the other figures use like reference numerals to identifylike elements. A letter after a reference numeral, such as “130 a,”indicates that the text refers specifically to the element having thatparticular reference numeral. A reference numeral in the text without afollowing letter, such as “130,” refers to any or all of the elements inthe figures bearing that reference numeral (e.g. “130” in the textrefers to reference numerals “130 a” and/or “130 b” in the figures).

The instrumentation analysis system 100 receives data comprising valuesof metrics sent by different development systems 120 (theinstrumentation analysis system 100 may also be referred to herein as ananalysis system or a data analysis system). A development system 120executes instrumented software, for example, application 130. Although,application 130 is shown in FIG. 1 as an example of instrumentedsoftware, the techniques disclosed herein are not limited to applicationsoftware but are applicable to other kinds of software, for example,server software, software executing on client devices, websites, and soon. Furthermore, a development system 120 comprises any computing systemthat is configured to execute instrumented software, whether or not itis used for development of new software. For example, the developmentsystem 120 may be a computing system used for testing purposes, stagingpurposes, or any production system executing in an enterprise.

The software executing on a development system 120 is configured to sendinformation generated as a result of instrumenting the software toinstrumentation analysis system 100. For example, the application 130may send values corresponding to various metrics as they are generatedto instrumentation analysis system 100. The application 130 may sendgroup values of metrics and send them periodically to instrumentationanalysis system 100. Different applications 130 may send the same metricor different metrics at different rates. The same application may senddifferent metrics at different rates. The application 130 sends data tothe instrumentation analysis system 100 by invoking applicationprogramming interface (API) supported by the instrumentation analysissystem 100.

A software program may be instrumented to add counters or gauges to theapplication. A counter comprises instructions that store a value that isincremented upon occurrence of certain event in the software. Thecounter may be used to determine the number of times a particular partof the code is executed, for example, a function or a method, aparticular branch of a conditional code, an exception, a loop, and soon.

Typically a counter value changes monotonically, for example, a countervalue may increase (or decrease) monotonically. For example, if thecounter tracks the number of times an event has occurred since thesystem started execution, the counter value increases each time theoccurrence of the event is detected by the system. Values of a countermay be compared to determine the change in the particular counter valueat two different points in time. For example, the number of times aparticular event occurs within a time interval between times t1 and t2may be determined by computing the change in a corresponding countervalue from t1 to t2. The APIs of the instrumentation analysis system maybe invoked by the application 130 to send the current value of thecounter to the instrumentation analysis system 100.

Following is an example of instrumented code of an application 130. Thefollowing instruction included in the code being instrumented creates acounter object for tracking count of an action or entities.

counter1=createCounter(source=“web1”, metric=“metric1”);

The above instruction creates a counter object and assigns it to thevariable counter1. The counter object is associated with a source “web1”and metric “metric1.” In an embodiment, the source and the metric valuesuniquely identify the data stream associated with the counter (or agauge). In other embodiments, more or fewer key value pairs may be usedto uniquely identify a data stream.

One or more of the values specified during creation of a counter arereceived when data corresponding to the counter is sent by theinstrumented code to the instrumentation analysis system 100.Embodiments allow the application 130 to be instrumented so as to reducethe amount of information sent with each data stream. This reduces theamount of overhead introduced in the application 130 as a result ofinstrumenting the code.

The instrumented code of application 130 may include instructions toupdate the counter value at various places in the code. For example, thecounter counter1 may be incremented by executing the instruction“counter1.increment( ).” The counter may be incremented to track variousactions or entities associated with the code. For example, the countermay be incremented whenever a particular function or method is called,the counter may be incremented whenever a particular branch of aconditional expression is executed, the counter may be incrementedwhenever an object of a particular type is created, for example, in aconstructor of an object. The increment instruction of the counter maybe called conditionally, for example, if a function is invoked with aparticular combination of parameters. The application 130 communicatesthe counter value to the instrumentation analysis system 100 by invokingan API of the instrumentation analysis system 100.

A gauge comprises instructions to measure certain runtimecharacteristics of the application 130, for example, heap size, numberof cache misses or hits, active memory used, CPU (central processingunit) utilization, total time taken to respond to a request, time takento connect to a service, and so on. A gauge may also be used to trackcertain application specific parameters or business related values, forexample, number of transactions, number of users, and so on. The gaugemay be invoked periodically based on an interval that is configurable.The value of the gauge is sent to instrumentation analysis system 100periodically.

The administration system 160 allows a privileged user, for example, asystem administrator to associate data streams with metadata. Theadministration system 160 comprises the administration application 170that provides a user interface for a system administrator to specify themetadata. The metadata comprises properties, for example, name-valuepairs. The instrumentation analysis system 100 receives metadatadescribing data streams and stores the metadata. The ability to specifymetadata describing data streams independently from the data receivedfrom each data stream provides several benefits in generating reportsbased on the data stream.

As an example, the instrumentation analysis system 100 can receivemodifications to metadata describing each data stream without requiringany modifications to the instrumented software of the application 130.As a result, the instrumentation analysis system 100 receivesspecifications of new reports and modifications to existing reports andgenerates results based on the new/modified reports without requiringthe developers to modify applications 130.

This provides for a new paradigm for instrumenting software since thedevelopers do not need to consider the types of reports that need to begenerated while adding instructions to instrument the software. Thedevelopers simply instrument their software to generate raw data thatcan be combined in various ways in the generated report. Systems andmethods for real time reporting based on instrumentation of software aredescribed in the U.S. patent application Ser. No. 14/800,677, filed onJul. 15, 2015 which is incorporated by reference hereby in its entirety.

Furthermore, the persons that are experts at generating the instrumentedsoftware can be different from the software developers. For example, anexpert at data analysis who is not a developer can define the metadatafor the data streams and generate reports without being involved in thedevelopment process. This is significant because the skills required foranalyzing data are typically different from the skills required fordeveloping software.

Furthermore, the instrumentation analysis system 100 can also receiveand process reports built on top of existing reports by composingexisting reports and adding new analytics functionality. Theinstrumentation analysis system 100 generates results of the new reportsand sends them for presentation in real-time as the instrumentationanalysis system 100 receives data streams from instrumented software.The instrumentation analysis system 100 generates these additionalreports and modifies existing reports without requiring anymodifications to the instrumented code of application 130.

Furthermore, the instrumentation analysis system 100 provides separationof the metadata describing the data streams from the data of the datastreams. Accordingly, the amount of data that needs to be transmittedfrom the development systems 120 to the instrumentation analysis system100 is reduced. Each application 130 transmits only the data values ofthe metrics and information identifying the metric. The metadatainformation is received separately from a source independent of the datasource of the data streams. Accordingly, any amount of metadata may beintroduced without increasing the amount of data of each data stream.

The reporting system 150 may be a client device. The reporting system150 includes a client application 140 that allows a user to interactwith the instrumentation analysis system 100. In an embodiment, theclient application 140 is an internet browser, which may include clientside code (e.g., Java Script) for accessing the instrumentation analysissystem 100. In other embodiments, client application 140 is aproprietary application developed for interacting with theinstrumentation analysis system 100.

The reporting system 150 can be a conventional computer system (e.g., adesktop or laptop computer), a tablet, or a device having computerfunctionality such as a personal digital assistant (PDA), a mobiletelephone, a smart phone or another suitable device. The reportingsystem 150 interacts with instrumentation analysis system 100 via anetwork. The network may comprise any combination of local area and/orwide area networks, using both wired and/or wireless communicationsystems. In one embodiment, the network uses standard communicationstechnologies and/or protocols.

The instrumentation analysis system 100 may be hosted on a computingsystem that includes one or more processors, memory, secondary storageand input/output controller. The computing system used for hosting theinstrumentation analysis system 100 is typically a server class systemthat uses powerful processors, large memory, and fast input/outputsystems compared to a typical computing system used, for example, as areporting system 150.

In an embodiment, data from several development systems 120 may beconsolidated, for example, by a server and the combined data sent to theinstrumentation analysis system 100. For example, an enterprise mayinstall a server that receives data stream internally from differentdevelopment systems 120 and sends the combined data in a batch form tothe instrumentation analysis system 100 periodically. This allowsefficiency of external communication from the enterprise. However thisconfiguration may result in delay in communicating information to theinstrumentation analysis system 100 and the corresponding delay inreporting data by the reporting system 150.

Associating Dimensions with Data Streams

A data stream may be identified by using a set of coordinatesrepresenting values of dimensions associated with data streams. Adimension refers to a property of data streams that can take one of aset of values. Each data stream may be associated with a value for adimension. For example, a dimension can be a source of a data stream ora metric name associated with a data stream. A source of a data streammay be identified by a server name, a service name, and so on. Examplesof metric names are cpu (central processing unit) load, cache misses,cache hits, and so on. A value of a dimension is also referred to as acoordinate value of the data stream. A coordinate value may berepresented as a metadata attribute stored in the metadata store 230.Given the two dimensions of source and metric, a data stream may beidentified by providing the two coordinates representing the source andthe metric, for example, (server1, cpu_load) or (server2, memory_usage).

A data stream may be characterized by multiple dimensions (i.e., morethan the two dimensions described above, i.e., source and metric name.)For example, if each server has multiple cpus, a dimension cpu_id may beincluded. Accordingly, each data stream obtained from a system may becharacterized by (source_id, cpu_id, metric_name), i.e., a sourceidentifier, a cpu identifier, and a name for the metric. Examples ofdata streams identified using three coordinates include (server1, cpu1,load), (server1, cpu2, load), (server2, cpu1, load), (server2, cpu2,load) and so on.

As another example of a dimension, a system may define customer name asa dimension. The name of the customer may be reported by theinstrumented software, for example, based on the configurationparameters of the instrumented software executing on a developmentsystem 120. The customer name may be specified for the instrumentedsoftware using a system property. The instrumented software includes thecustomer name when it identifies a data stream associated with thatparticular customer. The ability to associate a data stream with acustomer allows the instrumentation analysis system to perform customerspecific analysis, for example, report on usages of systems for eachcustomer, identify customers reporting more than a threshold number oferrors and so on.

A data stream may be obtained from instrumented software or may begenerated as a result of execution of blocks of a data stream languageprogram within the instrumentation analysis system. A data stream mayalso comprise data stored in the instrumentation analysis system, forexample, in a data store (such as a time series data store 260 describedherein.)

System Architecture of the Instrumentation Analysis System

FIG. 2 shows the architecture of a system for executing a data streamlanguage program for processing data streams received from instrumentedsoftware, according to an embodiment. The instrumentation analysissystem 100 includes an interface module 210, a quantization module 240,metadata module 220, metadata store 230, a data point routing module250, an analytics engine 270, a user interface manager 280, a datastream language processor 200, a time series data store 260, andsoftware bus 290. In other embodiments, the instrumentation analysissystem 100 may include other modules not described herein. Functionalityindicated as provided by a particular module may be implemented by othermodules instead.

The interface module 210 receives requests from external systems, forexample, development systems 120 that communicate with theinstrumentation analysis system 100. The interface module 210 supportsvarious application programming interfaces (APIs) that external systemscan invoke. The interface module 210 can receive and process dataprovided by applications 130 that are instrumented using functionalityprovided by different vendors, so long as the instrumented code sendsthe information in a format that can be processed by the interfacemodule 210.

The interface module 210 receives data in the form of data streams fromone or more development systems 120. In an embodiment, the interfacemodule 210 receives data and represents the incoming data as tuples.Accordingly, each data stream is represented as a plurality of tuples,each tuple representing a data point. A tuple of data received by theinterface module 210 comprises various elements. A tuple of dataincludes a metric identifier, for example, a name of the metriccorresponding to the tuple and a value of the metric. The tuple of datareceived may further comprise other elements, for example, a timestampcorresponding to the time that the data was captured by the application130 sending the data, one or more properties associated with the data.

In an embodiment, the timestamp associated with a tuple represents thetime that the data value was received by the instrumentation analysissystem 100. The properties associated with the data may be provided inthe form of name, value pairs. These properties may provide additionalinformation describing the data received, for example, informationdescribing the source of the data such as a host name, server name,device name, or service name associated with the source, a method orfunction name associated with the data, an application instanceidentifier, and so on.

In an embodiment, the interface module 210 generates and assigns anidentifier to records received by the interface module 210. Theidentifier is referred to herein as a time series identifier (alsoreferred to herein as a TSID or tsid). A unique time series identifieris assigned to all tuples matching a metric name and a set of propertiesreceived with the tuple. Accordingly, a tuple (metric name, properties,metric value, timestamp) gets mapped to a tuple (tsid, metric value,timestamp). For example, if a tuple provides a metric name m1, and ahostname h1, all tuples with metric name m1 and hostname h1 are assignedthe same time series identifier. Accordingly, the tsid uniquelyidentifies all tuples of a data stream received by the instrumentationanalysis system 100.

The quantization module 240 processes data values received so as totransform an input time series of data in which data is available atarbitrary time intervals to a time series in which data is available atregular time intervals. For example, the data values received in aninput time series may occur at irregular interval, however, thequantization module 240 processes the data of the time series togenerate a time series with data occurring periodically, such as everysecond, or every 5 seconds, or every 15 seconds, and so on. This processis referred to herein as quantization of the time series. In anembodiment, the interface module 210 creates multiple threads orprocesses, each thread or process configured to receive datacorresponding to a data stream. Each thread or process invokes thequantization module 240 to perform quantization of the data received foreach data stream for each time interval. Systems and methods forquantization of data streams of instrumented software are described inthe U.S. patent application Ser. No. 14/800,679, filed on Jul. 15, 2015which is incorporated by reference hereby in its entirety.

The metadata module 220 receives and stores metadata informationdescribing various data streams received from the development systems120. In an embodiment, the metadata stored in the metadata module 220 isreceived from a user, for example, a system administrator interactingwith the instrumentation analysis system 100 using the administrationsystem 160.

The metadata may be represented as name-value pairs. In an embodiment,the metadata is represented as metadata objects, each object defining aset of properties that may be represented as name-value pairs. A set ofdata streams may be associated with the metadata object. Accordingly,all properties represented by the metadata object are associated witheach data stream that is associated with the metadata object.

The metadata datastore 230 stores the metadata objects and theirassociations with the data streams. The metadata datastore 230 stores anidentifier (ID) for each metadata object and the properties representedby the metadata object. In an embodiment, each data stream is associatedwith a time series identifier that uniquely identifies the data stream.The metadata datastore 230 stores an index that maps each metadataobject to a set of time series identifier values. The metadata store 230may receive instructions to modify a metadata object. For example, themetadata store 230 may receive instructions to modify, add or deletesome properties represented by a metadata object. Alternatively, themetadata store 230 may receive instructions to modify the mapping from ametadata object to a data stream. For example, the metadata store 230may receive instructions to associate a data stream with a metadataobject or delete an association between a metadata object and a datastream.

In an embodiment, the metadata store 230 is represented as a relationaldatabase but may be represented as any other type of database or datastore. For example, the metadata store 230 may be a relational databasestoring tables that map metadata object IDs to time series IDsidentifying data streams. Other database tables may store the propertiesassociated with each metadata object as a mapping from metadata objectID to each property represented as a name-value pair.

The user interface manager 280 renders the user interface for allowingusers to specify the parameters of a data stream language program and topresent results of execution of the data stream language program. Theuser interface manager 280 may display real-time results of a datastream language program as one or more charts that are periodicallyupdated as the data of the data streams is received. The user interfacemanager 280 also presents a user interface that allows users to specifya data stream language program visually rather than textually. Examplesof screenshots of user interfaces presented by the user interfacemanager 280 are described herein.

The time series data store 260 stores data received from varioussources, for example, development systems 120. The time series datastore 260 is also referred to herein as time series database (or TSDB.)In an embodiment, the time series data store 260 also stores the timeseries data after the data is quantized. The time series data store 260may also store rollup data for each time series. The time series datastore 260 also stores results of various analytics requests, forexample, results of various reports requested by user. The analyticsengine 270 computes results for certain reports, for example, movingaverages over intervals of time by combining data stored in the timeseries data store 260 with new data obtained as data stream from varioussources.

The software bus 290 provides a mechanism for modules of theinstrumentation analysis system 100 to provide data of data streams toother modules of the instrumentation analysis system 100. A data streamlanguage program may send a data stream to the software bus 290. Othermodules, for example, fetch module 320, find module 310, window module380, and so on can read the data from the software bus 290 and performfurther processing on the data. For example, a data stream output of adata stream language program published on the software bus 290 may beidentified by a find block of another data stream language programexecuting as a job.

The data stream language processor 200 executes programs specified usingthe data stream language. The data stream language processor 200receives a data stream language program, parses the data stream languageprogram to validate the program. The data stream language processor 200generates a representation of the data stream language program andexecutes the data stream language program using the representation.

The requests specified using the data stream language is a query basedon the metadata associated with data received from various developmentsystems 120. The data stream language supports various types of analyticfunctions, for example, aggregations and transformations. The datastream language provides the ability to compose various functionsincluding aggregations and transformations in various ways. In anembodiment, the data stream language processor 200 parses programsspecified using the data stream language, generates an executablerepresentation of the program, and executes the generatedrepresentation.

Data Stream Language

A program specified using the data stream language comprises units ofcomputation called blocks. Each block is associated with a particularprocessing or computation performed by the data block. Each block mayalso have one or more input ports and one or more output ports. A blockreceives input via an input port, performs certain computation using thedata and sends the result of the computation to the output port. Thisprocess is repeated at a pre-specified periodicity. Accordingly, aninput port acts as a mechanism to provide data to the block and anoutput port acts as a mechanism to output data of the block.

In an embodiment, each block is associated with a type of the block. Thetype of the block determines the computation performed by the block. Thetypes of blocks supported by the data stream language include a findblock, a fetch block, a statistical computation block, a thresholdblock, and so on. A block may be associated with certain configurationparameters. For example, a find block may take an expression as input. Adata stream language program includes instances of a type of block. Forexample, a find block with a particular search expression is an instanceof the find block that is included in a data stream language program.

In an embodiment, an input port of a block is identified with character“?” and an output port is identified with character “!”. Otherembodiments may identify the input/output ports using other syntax. Forexample, if a block B1 has input ports in1 and in2, a specific inputport (say in2) may be identified as “B1?in2”. Similarly, if block B1 hasoutput ports out1 and out2, a specific output port (say out2) can bespecified as “B2!out2”. If a block has a single input/output port, thedata stream language program may not identify the port. For example, ifblock B2 has a single input port, the input port may be referred to as“B2”. Similarly, if block B2 has a single output port, the output portmay be referred to as “B2”.

Two blocks may be connected by specifying that the output of one blockis provided as input of the other block. Accordingly, a data streamlanguage program can be considered a network of blocks. In anembodiment, the connection between two blocks is specified using anarrow between the two blocks. For example, if B1 and B2 both have asingle input port and a single input port, “B1→B2” specifies that theoutput of B1 is provided as input of block B2. Similarly, if B1 has twooutput ports out1 and out2 and B2 has two input ports i1 and in2, theout1 port of B1 may be connected to the in2 port of B2 by the expression“B1!out1→B2?in2”.

The data stream language processor 200 may execute multiple jobs basedon a data stream language program. Each job may be associated with astart time, an end time, and a periodicity. Accordingly, the job isexecuted from the start time until the end time at intervals specifiedby the periodicity. The periodicity specifies the rate at which data isprocessed by the data stream language program. A user may specifydifferent jobs for execution based on the same data stream languageprogram, each job associated with different start time, end time, andperiodicity.

FIG. 3 shows the architecture the data stream language processor forprocessing blocks of data stream language programs, according to anembodiment. As shown in FIG. 3, the data stream language processor 200includes modules for processing various types of blocks of the datastream language. Accordingly, the data stream language processor 200includes a find module 310, a fetch module 320, a computation module330, a threshold module 340, a publish module 350, a grouping module360, a window module 380, a data stream metadata generator 370, and acustomized block module 390. Other embodiments may include more or lessmodules than those shown in FIG. 3. Certain modules are not illustratedin FIG. 3, for example, a parser. The details of each module are furtherdescribed herein along with details of the types of blocks processed byeach module.

The find module 310 executes the find block to identify a set of datastreams for processing by the rest of the data stream language program.The fetch module 320 fetches data from the identified data streams andprovides the data for processing by subsequent blocks of the data streamlanguage program. The computation module 330 performs statisticalcomputations specified in the data stream language program, for example,mean, median, sum, and so on. The threshold module 340 compares data ofan incoming data stream with a threshold value to determine if theincoming data exceeds certain bounds. The threshold value specified forcomparison may dynamically change, for example, a threshold value may bespecified as a one hour moving average of the input data stream scaledby certain factor. The publish module 350 executes the publish blockthat provides the output of the blocks preceding the publish block tovarious receivers including a user interface (e.g., a dashboard) forpresenting the results, for storing in a database, or for providing toother blocks for further processing. The grouping module 360 performsgrouping of data of input data streams to generate a set of result datastreams corresponding to each group. The groups may be based on one ormore attributes specified with the grouping command, for example, groupsof data streams from each data center. The data stream metadatagenerator 370 generates metadata representing result data streamsgenerated as a result of executing data stream language programs andstores the metadata in the metadata store 230 for allowing othercomponents of the instrumentation analysis system 100 to use the resultdata stream. The customized block module 390 processes user definedblocks (customized blocks) in a data stream language program.

Example Data Stream Language Program

FIG. 4 shows an example of a data stream language program forillustrating features of the data stream language, according to anembodiment. FIG. 4 represents the data stream language program in termsof blocks. The data stream language program shown in FIG. 4 can bespecified as follows.

find(“source:analytics*”)→fetch

-   -   →groupby(“datacenter”)    -   →stats!mean    -   →publish

The first block of the above data stream language program is a findblock 410 that takes a string parameter that specifies a searchexpression. The find block finds a set of data streams received by theinstrumentation analysis system 100 that satisfy the search expression.For example, the find block 410 takes search expression “source:dev”that identifies all data stream that the “source” metadata attributevalue “dev.” For example, an enterprise may associated all developmentsystems with source value “dev.” The output of the find block isprovides as input to a fetch block 420.

The fetch block 420 retrieves data from the data streams identified bythe find block. The fetch block receives data at a pre-specifiedperiodicity. The fetch block may receive real time data of data streamsreceived by the interface module 210 and quantized by the quantizationmodule 240. The fetch block 420 may also receive data of data streamsstored in the time series data store 260. The output of the fetch block420 is provided as input to the groupby block 430.

The groupby block 430 takes names of one or more attributes of datastreams as input. The groupby block 430 groups the data streams by thespecified attributes. As shown in the example above, the groupby block430 takes a “datacenter” attribute as input and groups the data streamsby their datacenter value. Accordingly, data of all data streams havingthe same data center is grouped together. The groupby block 430 outputsa data stream corresponding to each value of data center. The output ofthe groupby block 430 is provided as input to the stats block 440 (whichis a type of statistical computation block).

The stats block 440 has multiple outputs, for example, mean, median,sum, and so on. Each output port provides values based on the type ofcomputation specified by the name of the output. The stats block 440computes the mean value for each group of data streams received as inputfrom the groupby block 430. Accordingly, the stats block 440 determinesthe mean of data received from data streams of each datacenter. As shownin FIG. 4, the mean output port of the stats block provides input to thepublish block 450.

The publish block 450 may be configured to publish the received input ona dashboard. The publish block may be configured to publish the data onthe software bus 290. The software bus 290 provides the data to allother modules of the instrumentation analysis system 100. The datastream language processor 200 executes the various blocks specifiedabove at a periodicity specified for the data stream language program.

Overall Process of Execution of a Data Stream Language Program

FIG. 5 shows the overall process of an instrumentation analysis systemfor processing data received from data streams based on a data streamlanguage program, according to an embodiment. The metadata module 220receives 510 metadata describing data streams. The metadata definitionis received independent of the data of the data streams themselves. Forexample, the data stream may simply provide tuples comprising a datavalue and a timestamp associated with the data value without providingany properties (for example, name-value pairs.) The metadata module 220receives the properties describing the data streams from a sourcedifferent from the source providing the data stream. For example, thedata streams are provided by instances of instrumented software that isexecuting on development system 120, whereas the metadata definition maybe provided by a system administrator via the administration system 160.

The analytics engine 270 receives 520 a data stream language programusing the metadata attributes describing data streams. The data streamlanguage program may represent a set of instructions provided to theinstrumentation analysis system 100 to generate reports describing theinstrumented software and provide the results in real-time, i.e., as thedata of the data streams is received.

The instrumentation analysis system 100 repeats the following steps asdata of various data streams is received by the instrumentation analysissystem 100 from various development systems 120. The interface module210 receives 530 data of different data streams. In an embodiment, theinterface module 210 waits for a fixed interval of time, for example, 1second or a few seconds and collects data received from different datastreams. In an embodiment, the quantization module 240 performsquantization of the data for each incoming data stream for each timeinterval. Accordingly, data from each data stream is aggregated into asingle value associated with the data stream for that time interval.

The analytics engine 270 executes 540 the data stream language programbased on the data of the data streams for the time interval. If the datais quantized for each data stream, the analytics engine 270 executes 540the data stream language program using the quantized values from eachdata stream. The data stream language program may include a publishblock that causes the analytics engine 270 to send the result(s) ofevaluation of the data stream language program for presentation, forexample, to a user interface.

The data stream language program may generate one or more data streams.The analytics engine 270 also stores the data streams generated as aresult of evaluation of the data stream language program, for example,in the time series data store 260. The analytics engine 270 creates oneor more new data streams (or time series) representing the results ofthe data stream language program. The new data streams are stored in thetime series data store 260. This allows the result of the data streamlanguage program to be used as input to other data stream languageprogram. For example, a data stream language program may generate datarepresenting the 95^(th) percentile of values received from a pluralityof data streams. The result of the data stream language program may bestored in the time series data store 260 as a new data stream. Theanalytics engine 270 may further execute another data stream languageprogram that computes a moving average value based on the generated datastream.

Quantization

The quantization of the input data streams simplifies processing of datausing the quantized data streams. For example, aggregate values based onmultiple data streams received can be determined for each time interval.This is performed by further aggregating data for a particular timeinterval across multiple data streams. In an embodiment, thequantization of an input data stream is performed at the end of eachtime interval so that the quantized data for the time interval isavailable for processing.

Furthermore, the instrumentation analysis system 100 stores thequantized data for individual data streams so that data across multipledata streams can be combined in various ways, for example, as specifiedin a request. In other words, a user may send a first request thatcombines data across a plurality of data streams in a first manner.Subsequently the user may send a new request for combining the dataacross different data streams in a different manner. For example, a usermay combine data across data streams to view aggregates computed overvarious data centers. However, subsequently the user may change therequest to view aggregates computed over different types ofapplications, different types of servers, different geographicalregions, and so on.

The instrumentation analysis system 100 may also receive a request inwhich the user modifies the set of data streams over which previous datastreams were aggregated. For example, the user may request theinstrumentation analysis system 100 to remove one or more data streamsfrom the set of data streams being aggregated and request an aggregatebased on the revised set. A user may send such a request to analyze theimpact of removing or adding a new server, application, or making anyother modification to the system configuration. The instrumentationanalysis system 100 keeps the quantized data stream's data and combinesthe quantized data streams data for different time intervals based onthese requests. Since the instrumentation analysis system 100 stores thequantized data streams data, the instrumentation analysis system 100 hasthe ability to efficiently combine data across data streams as needed.

The instrumentation analysis system 100 can combine data across datastreams to perform moving aggregate calculations across multiple datastreams. The instrumentation analysis system 100 may continuouslycompute any moving aggregate value across a given length of timeinterval, for example, one hour moving average, a 15 minute movingaverage, and so on.

The quantization module 240 aggregates the values of the input datastreams for each time interval and generates an aggregate value for thetime interval. Accordingly, the quantization module 240 receives a datastream in which data values can occur after arbitrary time intervals.The quantization module 240 processes the input data stream to generatea data stream in which the data is available at regular time intervals.The details of the quantization module 240 are further described herein.

The quantization module 240 receives information describing the type ofvalue received in the data streams, for example, whether the value is acount of certain action or entities, whether the value was obtained byan aggregation of certain value, whether the value represents amaximum/minimum value of a given set of values, and so on. The type ofvalue of the data stream describes the types of operations performed toobtain the value. The quantization module 240 stores a mapping from thevarious types of values of the data stream to the type of operationperformed on the input values of the data stream for an interval toobtain the result value representing the time interval.

In an embodiment, the quantization module 240 includes a buffer forstoring data values that are received as input for a particular timeinterval. The buffer of the quantization module 240 uses a datastructure that can store arbitrary number of values since the number ofvalues received in a time interval is not known in advance and canchange from one time interval to another. For example, the quantizationmodule 240 may use a list data structure or a stack data structure forstoring the values of the input data stream.

The quantization module 240 collects the data values of the data streamreceived for each time interval. The quantization module 240 tracks thetime. When the quantization module 240 determines that the end of thecurrent time interval is reached, the quantization module 240 processesall the data values received in the time interval to determine theaggregate value representing the time interval. The quantization module240 subsequently clears the buffer used for representing the inputvalues and uses it to store the values for next time interval. In anembodiment, the quantization module 240 uses multiple buffers so thatwhile the data of a previous time interval stored in a buffer is beingprocessed, new data for the next time interval can be stored in anotherbuffer.

FIG. 6 illustrates the process of quantization of the data streamsreceived from instrumented software, according to an embodiment. FIG. 6shows time axes 620 a and 620 b, each representing a time line withseries of data values. The time axis 620 a shows the data values of theinput data stream 600 and time axis 620 b shows the values of thequantized data stream 610 generated by the quantization module 240.

As shown in FIG. 6, four data values D11, D12, D13, and D14 are receivedin the time interval I1 (representing the time from T0 to T1); two datavalues D21 and D22 are received in the time interval I2 (representingthe time from T1 to T2); and three data values D31, D32, and D33 arereceived in the time interval I3 (representing the time from T2 to T3).Each time interval between Tm and Tn may be assumed to include the starttime point Tm (such that the end time point Tn is included in the nexttime interval). Any other interpretation of the time interval between Tmand Tn may be used, for example, the end time point Tn included in thetime interval and the start time point Tm included in the previous timeinterval.

The quantization module 240 processes the data values of each timeinterval to generate the corresponding result value shown in the timeaxis 620 b. For example, the quantization module 240 aggregates thevalues D11, D12, D13, and D14 received in the time interval I1 togenerate the value D1 shown in time axis 620 b; the quantization module240 aggregates the values D21 and D22 received in the time interval I2to generate the value D2 shown in time axis 620 b; and the quantizationmodule 240 aggregates the values D31, D32, and D33 received in the timeinterval I3 to generate the value D3 shown in time axis 620 b.

The type of operation performed to aggregate the input values of thedata streams depends on the type of data represented by the input datastreams. If each tuple of the input data stream is a count of certainvalue, for example, a count of actions performed by the software, thequantization module 240 aggregates the input values to determine theoutput data stream value for each time interval by adding the counts. Ifeach tuple of the input data stream received is a minimum (or maximum)of a set of values, the quantization module 240 aggregates the inputvalues for a time interval to determine the output value for that timeinterval by determining the minimum (or maximum) of the input values forthe time interval. If each tuple of the input data stream received is anaverage of a set of values, the quantization module 240 aggregates theinput values associated with the time interval to determine the outputdata stream value for each time interval by determining an average ofthe input values of the time interval. If each tuple of the input datastream received is the last available value of the metric at that pointin time, the quantization module 240 aggregates the input values for thetime interval to determine the output value for that time interval bysimply using the last value of the data stream.

Metric Data Streams and Event Data Streams

In an embodiment, the instrumentation analysis system 100 supports twotypes of data streams, metric data streams and event data streams. Anevent typically refers to an exceptional condition that occurs in asystem, for example, load exceeding certain threshold values or memoryusage exceeding certain threshold values. An event may also refer toparticular actions performed in a system, for example, by a systemadministrator of a development system 120. A metric data streamcomprises data representing values of metrics that may be obtained frominstrumented software or derived from metric data streams obtained frominstrumented software. A data stream referred to herein is a metric datastream unless indicated otherwise. A metric data stream is also referredto as a metric time series and an event data stream is also referred toas an event time series.

A metric data stream comprises data points represented using: a datastream identifier, a time stamp value, and a data value. The data streamidentifier identifies the data stream to which the data point belongs.The time stamp value associates data point with a time, for example, thetime at which the data point was reported or the time at which the datapoint was received by the instrumentation analysis system 100. The datavalue is the value of the metric being reported, for example, the valuerepresenting the CPU load in a server at a particular time, or a measureof memory usage in a server at a particular time. A metric time seriestypically provides a large amount of data to the instrumentationanalysis system, for example, each data stream may report several datapoints each second and there may be a large number of data streams foreach enterprise.

An event data stream comprises data points represented using: a datastream identifier, a timestamp value, and one or more key value pairsdescribing an event. The data stream identifier and the timestamp valuesof an event data stream are similar to the metric data stream. However,events typically occur with less frequency compared to data points ofmetric data stream. For example, an event may represent an actionperformed by a system administrator, such as starting a maintenancewindow. The key value pairs of the event describe the event, forexample, the name of the system administrator that started themaintenance window, the purpose of the maintenance window, the scope ofthe maintenance window and so on. Events typically occur at an irregularrate, for example, events may be reported by some system but not others,events may occur once and may not occur for significant amount of time,and so on. As a result, the amount of information stored with events canbe large.

An event may also describe certain specific conditions occurring in asystem, for example, certain metrics displaying certain characteristic.As an example, an event may be reported if the cpu load or memory usageof a server exceeds certain threshold values. These events are generatedby the instrumentation analysis system 100 as a result of execution ofdata stream language programs.

The instrumentation analysis system 100 treats event time series thesame way as metric time series in terms of processing the data. Forexample, the instrumentation analysis system 100 allows real timereporting of information based on either type of data streams. Theinstrumentation analysis system 100 allows an event data stream to becompared with a metric data stream to allow a user to correlate the two.For example, a report may be generated that overlays a metric datastream with an event data stream indicating the metric values when theevent was generated.

Dynamic Selection of Data Streams for a Data Stream Language Program

The find block allows dynamic selection of data streams is input for adata stream language program. The find block specifies a searchcondition for identifying data streams. In an embodiment, the searchcondition is an expression based on attributes (or metadata tags)describing data streams. These attributes may be received as part of thedata stream or associated with the data stream, for example, as metadataadded to the instrumentation analysis system 100 and stored in themetadata store 230. The data streams identified by executing the searchcondition are provided as input to the subsequent block of the datastream language program.

The data stream language processor 200 may evaluate the search conditionof the find block periodically, thereby reevaluating the set of datastreams provided as input to the data stream language program. As aresult, the set of data streams provided as input to the data streamlanguage program is dynamically changed. For example, a developmentsystem 120 may add new servers, start or stop services, or reconfigureexisting services. Furthermore, new development system 120 may send datastreams to the instrumentation analysis system 100. As a result, the setof data streams received by the instrumentation analysis system 100changes dynamically.

The search condition of the find block may be used to identify a set ofdata streams based on characteristics of the data stream. For example,search conditions may be used to identify services belonging to aparticular data center, services corresponding to a particularapplication, services associated with an organization that may be spreadacross multiple data centers, services running a particular version of asoftware (say operating system, or an application having certain patch.)The type of search conditions specified for a find block depends on thetype of metadata tags defined for the data streams and stored in themetadata store 230.

The search condition of a find block is evaluated over all data streamsreceived from external systems such as development systems as well asdata streams generated within the instrumentation analysis system 100,for example, as intermediate or final results of data stream languageprograms. For example, as described herein, intermediate or finalresults of data stream language programs are represented as first classcitizens that are treated the same as data streams received fromdevelopment systems 120. Accordingly, when the search condition of afind block is evaluated, the result may include data streams receivedfrom developments systems 120 as well as data streams internallygenerated within the instrumentation analysis system 100.

Following are a few examples of search conditions specified for findblocks. Assume that a user wants to find load on analytics servers andthe analytics servers are named analytic1, analytic2, analytic3, . . . ,and analyticN. The set of analytics servers can be identified by using afind block find(“source:analytic*”) that specifies the search conditionas all data streams with metadata tag value satisfying the regularexpression “analytic*”.

The search condition may be a logical expression. For example, the findblock find(“source:databank* AND metric:numCacheHits”) finds all datastreams having source attribute of the form “databank*” and the metricname numCacheHits. Accordingly, the data stream language program withthis find block is evaluated for all data streams providing the metricnumCacheHits from sources identified as “databank*”. Similarly, the findblock find(“source:databank* AND metric:numCacheMisses”) finds all datastreams providing the metric numCacheMisses from sources identified as“databank*”. As another example, the find block find(“source:zk* ANDsmetric:cpu AND region:ore1”) finds all data streams having source nameof the form “zk*” from region “ore1” having metric “cpu”.

The find block may be associated with configuration parametersspecifying one or more of a start time, a stop time, and a periodicity.The periodicity of the find block may be different from the periodicityof job of the data stream language program to which the find blockbelongs. This is so because the rate at which the set of data streamsmay be the different from the rate at which the user would like the datato move through the data stream language program. For example, a usermay determine that the set of data streams doesn't change often and thesearch string may be evaluated once every hour or so whereas theperiodicity of the job is 1 minute. Accordingly, the user may specifydifferent values of periodicity for the find block and the data streamlanguage program.

In an embodiment, the evaluation of the find block is not based on afixed periodicity but triggered by certain events that occur in theinstrumentation analysis system 100. For example, the evaluation of thefind block is triggered by any update in the metadata. An update in themetadata may cause the result of the find block to change, resulting ina different set of input data streams being processed by the data streamlanguage program based on the find block. In an embodiment, theinstrumentation analysis system 100 associates a find block withspecific portions of metadata. In an embodiment, if the find block isbased on certain metadata attributes, any change associated with thosemetadata attributes triggers the execution of the find block. Forexample, if the find block evaluates to true for all data streams fromregion “xyz”, the evaluation of data streams is triggered by anyaddition or deletion of data streams to the region “xyz.” The additionor deletion of data streams to other regions may not trigger theexecution of the find block. The instrumentation analysis system 100analyzes and identifies sets of metadata attributes associated with eachfind block. The instrumentation analysis system 100 detects if a changein metadata occurs that is associated with the set of metadataattributes associated with a find block. If the instrumentation analysissystem 100 detects that a change in metadata has occurred that isassociated with the set of metadata attributes associated with a findblock, the instrumentation analysis system 100 reevaluates the findblock. In an embodiment, the instrumentation analysis system 100re-evaluates the find block if it detects that properties associatedwith a data stream have changed. In an embodiment, the find block isre-evaluated if the definition of find-block is modified.

In an embodiment, the find blocks are re-evaluated when there arechanges in data streams. For example, if new data streams are detectedby the instrumentation analysis system 100 or if the instrumentationanalysis system 100 determines that a data stream is inactive, theinstrumentation analysis system 100 re-evaluates the find block. Thedata streams generated may be data streams received from externalsystems such as development systems 120 or the data streams may begenerated by an intermediate or final result of data stream languageprogram. For example, as described herein, intermediate or final resultsof data stream language programs are represented as first class citizensthat are treated the same as data streams received from developmentsystems 120. Accordingly, addition, deletion, or modification ofmetadata of these data streams also causes the find block to bere-evaluted.

FIG. 7 illustrates selection of a set of data streams by a find blockfor providing input to a data stream language program, according to anembodiment. As shown in FIG. 7, the find block 710 a has a searchcondition specified by search string “datacenter:east*.” The find module310 of the data stream language processor 200 identifies all datastreams for which the “datacenter” metadata tag (or attribute) satisfiesthe regular expression “east*”.

FIG. 7 shows a set 740 a of data streams received by the instrumentationanalysis system 100 including data streams having datacenter tag valuescentral_dev, east_dev, east_qa, west_dev, and north_dev. The find module310 determines that the data streams with data center tag valueseast_dev and east_qa satisfy the search condition of the find block 710a. The find module 310 provides the set of identified data streams 750 afor the subsequent block 730 a of the data stream language program.

The set of data streams provided as input to the rest of the data streamlanguage program depends on the search condition associated with thefind block 710. For example, the find block 710 b has search condition“datacenter:*dev” which is different from the search condition of thefind block 710 a. The find module 310 of the data stream languageprocessor 200 processes the search condition of the find block 710 b byidentifying all data streams for which the “datacenter” metadata tag (orattribute) satisfies the regular expression “*dev”.

FIG. 7 shows a set 740 b of data streams received by the instrumentationanalysis system 100 including data streams having datacenter tag valuescentral_dev, east_dev, east_qa, west_dev, and north_dev. In thisexample, set 740 b has elements same as set 740 a. The find module 310determines that the data streams with data center tag valuescentral_dev, east_dev, west_dev, and north_dev satisfy the searchcondition of the find block. The find module 310 provides the set ofidentified data streams 750 b for the subsequent block 730 b of the datastream language program.

FIG. 7 illustrates dynamically determining the set of data streamsprocessed by a data stream language program by the data stream languageprocessor 200. The set of data streams processed by the data streamlanguage is determined based on the search condition of the find block710 and the currently available data streams received by theinstrumentation analysis system 100.

In an embodiment, the find block is associated with a schedule such thatthe find module 310 of the data stream language processor 200 executesthe find block according to the schedule. For example, the find blockmay be associated with a periodicity such that the find module 310executes the find block at a rate determined based on the periodicity.Accordingly, the find module 310 waits for a time interval based on theperiodicity and reevaluates the set of data streams satisfying thesearch condition of the find block. This process is repeated (until thetime reaches an “end time” value associated with the find block.)

FIG. 8 illustrates dynamic changes to the set of data streams providinginput to a data stream language program as a result of periodicre-evaluation of the find block, according to an embodiment. As shown inFIG. 8, the search condition of the find block is evaluated at time T1and again at time T2 resulting in different sets 850 of data streamsbeing identified for processing by the data stream language program.FIG. 8 illustrates re-executing the find block at two different timepoints.

At time T1, the instrumentation analysis system 100 receives a set 840 aof data streams with datacenter tag values central_dev, east_dev,east_qa, west_dev, and north_dev (note that there may be multiple datastreams with the same datacenter tag values). The find module 310evaluates the find block 810 a with search condition “datacenter:east*”.Accordingly, the find module 310 identifies a set 850 a of data streamswith datacenter tag values east_dev and east_qa. The data streamlanguage processor 200 provides the set 850 a of data streams identifiedto the subsequent block 830 a of the data stream language program.

The find module 310 re-evaluates the find block at time T2. At time T2,the instrumentation analysis system 100 receives a set 840 a of datastreams with datacenter tag values central_dev, east_dev, east_prod,west_dev, and north_dev. Accordingly, the find module 310 identifies set850 b of data streams with datacenter tag values east_prod and east_qa.

Compared to the set 850 a identified at time T1, the set 850 b includesa new data stream with datacenter tag east_prod and lacks the datastream with datacenter tag east_qa. The data stream language processor200 provides the set 850 a of data streams identified to the subsequentblock 830 a of the data stream language program. Accordingly, eachsubsequent evaluation of the set 850 of data streams based on the samesearch condition of the find module may result in a different set ofdata streams being provided to the subsequent blocks 830.

The ability to dynamically change the set of data streams that areprocessed by a data stream language program allows the data streamlanguage program to adapt to a dynamically changing environment thatprovides input to the instrumentation analysis system. For example, anenterprise may add/remove servers to a data center, add new datacenters, add/remove/modify services, change services to execute softwareinstrumented in different ways and so on. The ability to specify the setof data streams processed by a data stream language program allows theinstrumentation analysis system to report data describing the enterpriseas it changes dynamically without having to modify the data streamlanguage program.

FIG. 9 shows the process for identifying a set of data streams forproviding input to a data stream language program using the find block,according to an embodiment. As shown in FIG. 9, the data stream languageprocessor 200 receives 900 a data stream language program forprocessing. The process illustrated in FIG. 9 is based on the assumptionthat the data stream language program has a find block followed by a setof blocks corresponding to the remaining data stream language program.

The find block is associated with a search string. The find module 310receives 910 the search string associated with the find block. The findmodule 310 parses 920 the search string to build a representation of thesearch condition corresponding to the search string, for example, aparse tree representation. The find module 310 identifies 930 a set ofdata streams corresponding to the search condition. The find module 310provides the set of identified data streams to the subsequent block ofthe data stream language program, for example, the fetch block. The datastream language processor 200 retrieves data from the data streamsidentified 930 based on the search condition and executes 940 theremaining data stream language program.

The steps of identifying 930 the set of data streams based on the searchcondition and executing 940 the remaining blocks of the data streamlanguage program are repeatedly executed by the data stream languageprocessor 200. The rate at which the steps 930 and 940 are repeated maybe different. For example, the step of identifying 930 the set of datastreams may be executed at a slower rate compared to the rate at whichthe remaining blocks of the data stream language program are executed.The rate of execution 940 of the remaining blocks of the data streamlanguage program and the rate of execution of the find block isspecified (for example, by a user) for a job corresponding to the datastream language program.

Retrieving Data from Data Streams for a Data Stream Language Program

In an embodiment, a data stream language program includes a fetch blockfor retrieving data from a given set of data streams. Typically thefetch block is placed after the find block in the data pipeline of thedata stream language program. In other words, the output of the findblock is provided as input to the fetch block. Accordingly, the fetchblock retrieves data from the set of data streams identified by the findmodule 310 as a result of processing a find block. The fetch module 320executes the fetch block.

FIG. 10 illustrates the process of retrieving data from data streams byexecuting a fetch block, according to an embodiment. Certain stepsindicated in FIG. 10 can be executed in an order different from thatindicated in FIG. 10. Furthermore, steps can be executed by modulesdifferent from those indicated herein.

The data stream language processor 200 receives the start time, endtime, and periodicity of execution of a job based on a data streamlanguage program. The fetch module 320 receives the set of data streamsfrom the find module 310 based on the search condition of the find blockof the data stream language program. The fetch module retrieves data andprovides it for execution to the subsequent block of the data streamlanguage program. The fetch module 320 performs the following steps forfetching data from data streams for each subsequent time interval.

The fetch module 320 identifies the next time interval and waits fordata to arrive during the time interval. The quantization modulegenerates multiple quantized data streams having different periodicitybased on data of each input data stream. For example, a quantized datastream Q1 may be generated with a periodicity of 5 second, anotherquantized data stream Q2 may be generated with a periodicity of 10second, another quantized data stream Q3 may be generated with aperiodicity of one minute, and so on. The fetch module 320 selects 1020the quantized data stream that has the largest periodic time intervalthat is smaller than the periodic time interval at which the data streamlanguage program is executed (as determined based on the periodicity ofthe data stream language program).

For example, if the size of the time interval at which the data streamlanguage program needs to be executed in 30 seconds based on theperiodicity of the data stream language program, the fetch module 320selects the quantized data stream Q2 having the periodicity of 10seconds. The quantized data stream Q3 is not selected because it has aperiodic time interval of 1 minute (i.e., 60 seconds) which is largerthan the time periodic time interval of the data stream language program(i.e., 30 seconds). The quantized data stream Q3 is not selected becauseit has a periodic time interval of 5 seconds which is not the largestperiodic time interval that is smaller than the periodic time intervalof the data stream language program (since it is smaller than theperiodic time interval of Q2 which is 10 seconds). The fetch module 320re-quantizes the selected quantized data stream to generate are-quantized data stream of periodicity 30 seconds (for example, byaggregating the data values of three data points of the quantized datastream that occur in the current 30 second time interval).

The fetch module 320 retrieves 1050 data from the time series data store260 if necessary to combine with the real time data being received fromdata streams. The fetch module provides 1060 the combined data to thesubsequent block, for example, a statistical computation block. Forexample, assume that a data stream language program publishes output toa screen and the start time of the job is indicated as negative (forexample, −1 hour). The data may be presented as a chart that presentsdata as it is received as well as past data for a selected timeinterval. For example, a user may select a one hour time window forpresenting the data on the chart. In this situation, if the chart wasrendered only based on the real time data received in the data streams,the chart would be empty when the instrumentation analysis system 100starts processing the data stream language program. The displayed chartwould slowly start filling from the right and would fill up thedisplayed window after an hour. This presents a user experience that isnot ideal. Ideally a user would like to see the full chart (with onehour of data) throughout the one hour that the chart is displayed fromthe beginning.

The fetch module 320 remedies the above situation by retrieving 1050data from the time series data store 260 for rendering the portion ofthe chart that occurs before the time for which the real time data fromthe data streams is available. For example, when the instrumentationanalysis system 100 starts processing the data stream language program,the fetch module 320 presents the data for rendering the entire chartusing the data obtained from the time series data store 260. As more andmore data is received from data streams, the fetch module 320 combinesthe data from the time series data store 260 with the real time datareceived.

As an example, after 10 minutes, the fetch module 320 sends forpresentation 50 minutes of data retrieved from the time series datastore 260 combined with 10 minutes of data received from data streams.Similarly, after 30 minutes, the fetch module 320 sends for presentation30 minutes of data retrieved from the time series data store 260combined with 30 minutes of data received from data streams, and so on.After more than 60 minutes of data of data streams is received, thefetch module 320 has sufficient data based on data streams that it cansend all the data for rendering the chart based on data received fromdata streams and does not have to combine the data from data stream withpreviously stored data of the time series data store 260.

The fetch module 320 may retrieve 1050 data from time series data store260 for combining with data received from data streams in othersituations, for example, for a window block. A window block provides asliding time window of a specified length (say tw) and performs acomputation of the data of the window (say average value) to determine amoving average over a one hour time window. In this situation, there isan initialization latency of time tw since the data from the datastreams is not available for a period of time tw to fill up the entirewindow. Accordingly, if the data stream language program starts at timet1, the data starting from time t1-tw is fetched from the time seriesdata store 260 to fill up the window to provide meaningful data for thewindow computation. At any time t0>t1, (while t0-t1 is less than tw),the fetch module 320 fills up the end portion of the window of lengtht0-t1 with real time data received from data streams and fills up thefirst portion (i.e., the remaining portion) of the window with dataretrieved from the time series data store 260.

If the data stream language program includes multiple windowscomputation, the fetch module 320 maintains data of the size of thelargest window that needs to be fetched by combining the data from thetime series data store 260 (if necessary) and the real time datareceived from data streams. The data maintained for the largest windowincludes data for smaller windows.

FIGS. 11A-C illustrate the process of combining data from the timeseries data store and data received in real-time from data streams formoving window calculations, according to an embodiment. The length ofthe moving window is assumed to be Tw. An example computation is anaggregation across data of a set of data streams, for example, averagevalue or a percentile calculation based on data received during themoving window across the set of data streams. The moving window is atime window that keeps shifting. In other words the size of the movingwindow stays constant but the window keeps advancing with time.

The number of data points that occur within the window may change overtime. The number of data streams processed may also change as the windowadvances, for example, due to introduction of new data streams or due tomodifications to metadata describing the data streams. For example, ifthe moving window is computing an average value of data across all datastreams from data center “east”, the number of data streams may changeover time if the data center “east” starts/stops services, introducesnew servers, or if the metadata describing data streams is modified toadd/remove the “datacenter=east” tag to/from certain data streams. Thedata stream language processor 200 periodically re-evaluates the set ofdata streams and also the set of data points that occur within thewindow and computes the aggregate value specified for the data pointsfrom the selected data streams.

FIG. 11A shows the scenario in which when a window computation isstarted, entire data of the window may be retrieved from the time seriesdata store 260. FIG. 11B shows that after some time (which is less thanthe time Tw, the length of the window), the fetch module 320 combinesdata from the time series data store 260 with real time data receivedfrom data streams. FIG. 11C shows that after a time greater than thelength of the window Tw, the fetch module 320 does not have to retrievedata from the time series data store 260 and can fill up the entirewindow with real time data obtained from the data streams.

As shown in FIG. 11A, T2 indicates the current time and given a windowof size Tw, time T1 represents the time point T2-Tw. Assume that thewindow computation starts at time T2. Accordingly, the window is in timerange T1 to T2. There is no data received from data streams at thispoint. The data for the entire window is retrieved from the time seriesdata store 260.

FIG. 11B shows that after some time, the current time is represented byT4 and the window has advanced to the time range T3 to T4. The real timedata is collected and used in the window calculation for the time rangeT2 to T4 since the real time data was collected since time T2. For thetime range T3 to T2, the fetch module 320 still uses data from the timeseries data store 260. The scenario shown in FIG. 11B applies for alltimes when the time range T4-T2 is less than Tw (in other words, for alltimes since T2 that is less than the size of the window).

FIG. 11C shows the scenario for times that are equal to or greater thanthe length of the window. In other words, if T5 is the current time,FIG. 11C applies for all times T5 such that T5-T2 is greater than orequal to the length of the window Tw. In these scenarios, the fetchmodule 320 has accumulated enough real-time data from data streams, thatthe fetch module 320 does not retrieve data from the time series datastore 260. In other words, the window computation is performed using allthe data received in real time from the data streams.

The scenario described in FIGS. 11A-C also applies for presenting datausing a chart (e.g., via a dashboard). The data from the time seriesdata store 260 is used to fill up the initial portion of a chart toavoid showing the chart filling up slowly as time advances. The abilityto fill up the chart with data from the time series data store 260provides for a better user experience since the user is presented with achart for the entire time window selected by the user.

Grouping Data Streams

FIG. 12 illustrates a process for grouping data of data streams togenerate a set of result data streams, according to an embodiment. Agrouping statement may be included in a data stream language program,for example, using the groupby block as shown in FIG. 4. The groupingstatement of a data stream language program specifies one or moremetadata attributes describing data streams. The groupby block isassociated with an aggregate computation that is performed for eachgroup of data streams.

The grouping module 360 receives 1210 one or more attributes describingdata streams. The attribute may be attributes received with the data ofthe data stream (for example, source name, and metric name) or metadatatags associated with the data stream by the metadata module 220 andstored in the metadata store 230. The grouping module 360 also receivesa particular computation to be performed for each group of data streams,for example, a computation determining an aggregate value based on dataof the data streams.

The data stream language processor 200 (and its component modules)perform the following computation for each time interval based on theperiodicity specified for the job executing the data stream languageprogram. The grouping module 360 identifies 1220 groups of data streamscorresponding to each distinct set of values of the one or moreattributes associated with the grouping command. For example, if theattribute specified with the grouping command is the “datacenter”attribute, the grouping module 360 identifies sets of data streams, eachset having a distinct value of the “datacenter” tag.

The grouping module 360 performs the following computations for each set(or group) of data streams identified. The grouping module 360 receives1230 data corresponding to each data stream of the set for thatparticular time interval. The grouping module 360 determines 1240 thevalue of the aggregate computation for the data from data streams ofeach group. For example, if the grouping is based on attribute“datacenter” and the computation specified is average, the groupingmodule 360 determines 1240 the average of data of all data streams for aparticular datacenter obtained for the given time interval. The groupingmodule 360 outputs 1250 the result of the computation for each group tothe subsequent block of the data stream language program.

As described in the process illustrated in FIG. 12, the groupingstatement (i.e., the groupby block) takes a set of data streams as inputand generates a set of result data streams. The grouping statement mayspecify grouping by a plurality of metadata attributes. The number ofresult data streams generated is equal to the number of distinctattribute values of the grouping attributes for which at least one datastream exists in the input set. In other words, a data stream isgenerated for each distinct combination of values of the groupingattributes if there are data streams in the input that have attributeswith that combination of distinct values.

FIGS. 13A-B shows an example scenario illustrating grouping of datastreams based on different metadata attributes describing the datastreams, according to an embodiment. FIG. 13A shows grouping of a set ofdata streams based on an attribute “dc” (representing data center.) Theinput set 1340 a of data streams includes a data stream with attributesdc=east and metric=cpuLoad, a data stream with dc=west andmetric=cpuLoad, a data stream with dc=north and metric=cpuLoad, a datastream with dc=west and metric=cacheMisses, and a data stream withdc=north and metric=cacheMisses. The grouping module 360 processes thegrouping block 1310 a that specifies groupby(“dc”) to collect datastreams from the input set 1340 a having the same attribute value forthe attribute dc. The input set 1340 a includes one data stream withdc=east, two data streams with dc=west, and two data streams withdc=north.

In an embodiment, grouping module 360 ignores distinct values of thegroup by attribute if there are no input data streams having thatcombination of values. Accordingly, the grouping module 360 does notgenerate any result data stream corresponding to these attribute values.For example, if the dc attribute can have other possible values, say,“north-east”, “south-west” and so on, and there are no input datastreams having these attribute values, the grouping module 360 does notgenerate any result data streams corresponding to the these distinctvalues of the metadata attributes.

Accordingly, as shown in FIG. 13, the grouping module 360 generatesthree result data streams, a first result data stream corresponding todc=east, a second result data stream corresponding to dc=west, and athird data stream corresponding to dc=north. Each result data streamscomprises data values generated by aggregating data from thecorresponding group of input data streams at a periodicity at which thegroup by block is executed (which is the periodicity at which the datastream language program is executed).

The grouping module 360 may generate a different set of result datastreams if the groupby block specifies a different attribute forgrouping. For example, FIG. 13B shows grouping of data streams based on“metric” attribute. The input set 1340 b has the same data streams asthe set 1340 a. The input data stream groups three data streams togenerate a results data stream corresponding to the metric=cpuLoad andanother result data stream corresponding to metric=cacheMisses.

FIG. 14 shows an example scenario illustrating dynamic changing ofresult data streams generated by a groupby block as a result of changesin input data streams over time, according to an embodiment. Forexample, the group by block shown in FIG. 13a may be executed at a laterpoint in time (for example, for a different time interval) when theinput set 1440 of data streams is different from the set 1340 a. Asshown in FIG. 14, the input set 1440 doesn't include any data streamwith attribute dc=east. Furthermore, the input set 1440 includes a datastream with dc=south. Accordingly, the grouping module 360 generates aresult set 1450 with three result data streams, a first result datastream corresponding to dc=west, a second result data streamcorresponding to dc=north, and a third data stream corresponding todc=south. Accordingly, the groups generated by the grouping module 360may dynamically change as the input set of data streams changes. Theinput set of data streams received from instrumented software executingin development system 120 may change for various reasons, for example,as a result of starting new development systems 120, adding/removingservices, or modifying metadata associated with the data streams in themetadata store 230.

Publishing Data Streams as First Class Citizens

According to an embodiment, a data stream language program includes apublish command (i.e., a publish block) that publishes one or more datastreams based on result of execution of a data stream language programby providing the data stream to other components of the instrumentationanalysis system 100. For example, a data stream generated by a datastream language program may be published to a user interface to bepresented as a real time chart or report. The generated data streams arerepresented as first class citizens. In other words, the generated datastreams are represented the same way as a data stream received from aninstrumented software of a development system 120 by the instrumentationanalysis system 100.

The generated data stream can also be used in the same way as a datastream received by the instrumentation analysis system 100 by othercomponents of the instrumentation analysis system 100. The generateddata streams can be associated with metadata attributes automatically bythe instrumentation analysis system 100 or by a system administrator viathe administration system 160. A find block of a data stream languageprogram can find the generated data stream similar to other data streamsreceived from external systems. Jobs executing other data streamlanguage programs can receive the generated data stream as input andprocess it. The data of the data stream can be presented via a userinterface and can be manipulated based on input received from the user,similar to any other data stream processed by the instrumentationanalysis system 100.

The data stream language processor 200 publishes result data streams onthe software bus 290. Any component of the instrumentation analysissystem 100 that can identify the data stream identifier for any resultdata stream (or any other data stream) can obtain the data of the datestream from the software bus 290. The software bus 290 may store data ofthe data streams published in memory to provide fast access to the data.

A data stream language program may generate multiple result data streamsfor publishing. For example, a data stream language program mayaggregate a metric (say, cacheMisses) grouped by data centers.Accordingly, an aggregate attribute (say, total cacheMisses) value isgenerated for each data center. The publish module 350 generatesmetadata describing each generated result data stream and stores themetadata in the metadata store 230. The publish module 350 associatesdata streams with information associated with the data stream languageprogram generating the data stream. Accordingly, the publish module 350analyzes the blocks of the data stream language program generating thedata stream and identifies information identifying the data stream fromblocks of the data stream language program.

The publish module 350 may generate metadata attributes describing adata stream based on attributes of the data streams received as input bythe data stream language program generating the published data stream.For example, if a data stream language program computes a moving averageof an input data stream, the publish module 350 associates metadataattribute values based on the input data stream with the published datastream as well. In this situation, the publish module 350 may use thesource name of the input data stream as the source name of the inputdata stream. If the published data stream is obtained by aggregating aplurality of input data streams, the publish module 350 may generate anattribute for the published data stream by aggregating attribute valuesbased on the input data streams (for example, by concatenatingcorresponding attribute values from the input data stream or byconcatenating substrings obtained by shortening attribute values fromthe input data stream.) For example, the source name of the result datastream may be obtained by concatenating source names of the input datastreams that are aggregated or by concatenating prefix strings of thesource names of the input data streams.

In an embodiment, a publish block is associated with a metric namecharacterizing the type of data being published. The publish module 350associates the metric name of the publish block with data streamspublished by the publish block. The data stream language processor 200also generates an identifier (called a time series identifier) forrepresenting each result data stream. The data of each result datastream is stored in the time series data store 260 and is available foruse by any component of the instrumentation analysis system.

If the publish block is not associated with a metric name, the publishmodule determines a metric name based on the input data streams receivedby the data stream language program that generated the data stream beingpublished. If the data stream language being published is generated froma single data stream, the publish module uses the metric name of thesingle data stream as the metric name of the published data stream. Ifthe data stream language being published is generated from a pluralityof data streams, the publish module generates a metric name for thepublished data stream based on the metric names of the plurality of datastreams, for example, by concatenating the metric names or substrings ofmetric names (e.g., prefixes or suffixes).

FIG. 15 shows a flowchart illustrating the process of publishing resultdata streams obtained by executing a publish block of a data streamlanguage program, according to an embodiment. The data stream languageprogram is assumed to include a publish block and one or more groupbyblocks. The publish block is assumed to be associated with a metricname. For example, the data stream language program may be as follows:

-   -   find(“source:analytics*”, “metric:load”))→        -   fetch( )→        -   groupby(“datacenter”)→        -   stats!mean→        -   publish(“dc_load”)

The above data stream language program includes a publish block thatspecifies a metric name “dc_load.” The data stream language program alsoincludes a groupby statement for grouping the input data streams bydatacenter.

The data stream language processor 200 identifies 1500 a publish blockin the data stream language program being processed. For example, if theabove data stream language program is being processed, the data streamlanguage processor 200 identifies 1500 the last block of the data streamlanguage program, i.e., publish(“dc_load”). The publish module 350determines 1510 a metric name associated with the publish block. Forexample, in the publish block of the data stream language program shownabove, the publish module 350 determines 1510 the metric name “dc_load”,associated with the publish block. The data stream language processor200 uses the metric name as a metadata attribute describing the resultdata streams.

The output of the publish block may include multiple result datastreams, for example, if the data stream language program includes agroupby block. The above example data stream language program maygenerate multiple result data streams, one for each datacenter, i.e.,one result data stream based on the statistical mean data valuesperiodically obtained from all data streams having a distinct datacenterattribute value. Other data stream language programs may includemultiple groupby blocks. However, the number of result data streamsgenerated by a data stream language program is determined by the lastgroupby block of the data stream language program.

The publish module 350 identifies 1520 the set of attributes of the lastgroupby block of the data stream language program. In the above example,the groupby(“datacenter”) block has a single attribute “datacenter” bywhich the data streams are grouped. However, a groupby block may includemultiple attributes for grouping the data streams. For example, thegroupby command groupby(“datacenter”, “region”) specifies two attributes“datacenter” and “region” by which the data streams are grouped. Thepublish module 350 uses distinct values of the identified set ofattributes for distinguishing result data streams generated by the datastream language program.

The data stream language processor 200 (and its component modules)performs the following steps for each result data stream. The publishmodule 350 identifies values of the identified attributes of the lastgroup by block that are associated with the result data stream. Thevalues of the identified attributes associated with the result datastream may be either received with the data stream or fetched from themetadata store 230 given the identifier of the input data streams of thegroupby block. If the input set of data streams includes data streamshaving different datacenter values, for example, “east”, “west”,“north”, “south” and so on, each result data stream output by thegroupby block (and the data stream language program if the groupby blockis the last groupby block of the data stream language program) isassociated with one of these datacenter values. If the groupby blockspecifies multiple attributes for grouping, each result data stream isassociated with a distinct set of values of the attributes specified thegroupby block for grouping.

The data stream metadata generator 370 generates 1540 the metadatadescribing the result data stream based on the values of the identifiedattributes associated with the result data stream and the metric nameassociated with the publish block. For example, if the groupby blockspecifies the data center attribute (with values “east”, “west”,“north”, “south”) and the metric name specified with the publish blockis cpu_load, the data stream metadata generator 370 associates eachpublished data stream with the metric name cpu_load and thecorresponding value of the datacenter attribute (associated with thegroup of data streams.) The data stream metadata generator 370 alsogenerates an identifier for the result data stream. The data streammetadata generator 370 stores 1550 the metadata comprising theattributes associated with the result stream in the metadata store 230.

The data stream language processor 200 periodically executes the datastream language program as specified by the periodicity of the datastream language program. The data stream language processor 200generates data for each result data stream when the data stream languageprogram is executed. The data stream language processor 200 stores 1560the generated data for each result data stream in association with theidentifier for the result data stream.

Anomaly Detection Using Threshold Blocks

The data stream language program supports threshold blocks that allowdata of a set of data streams to be compared against threshold values.The data streams being compared may be data streams received by theinstrumentation analysis system 100 from instrumented software ofdevelopment systems 120 or data streams obtained as a result ofexecution of one or more blocks of data stream language programs. Thethreshold block includes a data port and a threshold port. The data portreceives one or more data streams representing data values. Thethreshold port receives one or more data streams representing thresholdvalues. The threshold block compares data values against thresholdvalues to determine whether the data values are within a range specifiedby the threshold values. In an embodiment, the threshold block includesmore than one threshold ports. For example, the threshold block mayinclude two threshold ports, a low threshold port and a high thresholdport. The threshold block determines whether the data values are belowthe threshold values received in the high threshold port and above thethreshold values received in the low threshold port.

The threshold block allows specification of a high threshold valueand/or a low threshold value. The threshold module 340 processes athreshold block by comparing data values received in incoming streamswith threshold values specified by the threshold block. The thresholdblock specifies a low threshold and a high threshold. The thresholdmodule 340 generates an event if the data values from the input datastreams received by the threshold block lie outside the bounds set ofthe high threshold value and/or the low threshold value. In other words,the threshold module 340 generates an event if data of a data streamexceeds a high threshold value or falls below a low threshold value. Thethreshold values may be fixed or dynamic. A dynamic threshold value isobtained as a result of execution of a data stream language program. Athreshold block may specify either one of low/high threshold or both.

The input to the threshold block may be a plurality of data streamvalues generated as a result of executing blocks of a data streamlanguage program, for example, a plurality of data streams obtained as aresult of grouping a set of input data streams. In this situation, thelow threshold or the high threshold is also specified as the output of adata stream language program that generates a plurality of data streams.The threshold module 340 matches data streams received by the input portof the threshold block with data streams received by the low/highthreshold ports. The threshold module 340 compares the data of the datastreams received by the input port with data of the data streamsreceived by the low/high threshold ports for each time interval (basedon the periodicity of the data stream language program) and takes actionbased on the comparison (e.g., sending events).

In an embodiment, the threshold block specifies a time duration and afraction value. For example, the threshold block may specify a timeduration T (say 5 minutes). The threshold module 340 generates an eventif the data of an input data stream is outside the specified thresholdvalues for more than the specified time duration T. For example, if thedata of an input data stream is higher than the high threshold for morethan T time units, the threshold module 340 generates an event. Asanother example, if the data of an input data stream is below the lowthreshold for more than T time units, the threshold module 340 generatesan event. The ability to specify the time duration ensures that theabnormal behavior of data of the data stream lying outside the thresholdboundaries persists for a significant amount of time and is not atransient behavior.

In an embodiment, the threshold block specifies a fraction value F (say0.8) along with the time duration T. The threshold module 340 generatesan event if the data of an input data stream lies outside the thresholdboundaries for more than the specified fraction of the time duration Tduring a window of the specified length T. Accordingly, the thresholdmodule 340 generates an event even if the data of an input data streamis not outside the threshold boundaries for the entire time duration T,so long as the data is outside the threshold boundaries for at least thespecified fraction of the time duration.

FIG. 16 shows an example of a data stream language program illustratinguse of a threshold block with fixed threshold values for data streamsgrouped by a particular attribute, according to an embodiment. The datastream language processor 200 receives the data stream languageprocessor shown in FIG. 16 and processes it.

The find module 310 executes the find block 1610 to identify a set ofdata streams that are input to the data stream language program 1600.The fetch module 320 executes the fetch block 1615 to fetch the data ofthe data streams at the periodicity specified for the data streamlanguage program. The grouping module 360 executes the groupby block1620 to group the data streams identified by the find block based on thedatacenter values into a set of data streams, each data stream of theset corresponding to a distinct datacenter value occurring in theidentified data streams. The computation module 330 executes the statsblock 1625 to determine the mean values corresponding to data from eachdata center. The computation module 330 provides the output of the statsblock 1625 as input to the in port of the threshold block.

The threshold module 340 compares data of each data stream input to thehigh threshold value of the threshold block 1630. As shown in FIG. 16,the high threshold value of the threshold block 1630 is a fixed value(i.e., the fixed value 6). Accordingly, if any data value of a datastream for any group (corresponding to a data center) exceeds the highthreshold value of 6, the threshold module 340 generates an event. Thethreshold module 340 provides the details of the data stream exceedingthe threshold value in the event as name value pairs. For example, thethreshold module 340 may provide details of the data center attributevalue corresponding to the data stream that exceeded the high thresholdvalue, the timestamp of the time at which the high threshold wasexceeded and so on. Since the threshold block 1630 does not specify alow threshold value, the threshold module 340 does not compare the dataof the data streams input to the threshold block 1630 to any lowthreshold value.

FIG. 17 shows an example of a data stream language program illustratinga threshold block with dynamically changing threshold values for datastreams grouped by metadata attributes, according to an embodiment. Thedata blocks providing input to the in port of the threshold block 1760of FIG. 17 are similar to the data blocks providing input to thethreshold block 1630 of FIG. 16. Accordingly, blocks 1710, 1715, 1720,1725 of FIG. 17 correspond to blocks 1610, 1615, 1620, and 1625 of FIG.16 respectively. However, the input to the high port of the thresholdblock 1760 receives a dynamically changing input. Furthermore, the highport of the threshold block 1760 receives a plurality of data streams asinput. The threshold module 340 matches the plurality of data streamsreceived by the high port of the threshold block 1760 with the pluralityof data streams received by the in port.

The fetch module 320 executes the fetch block 1730 to fetch the data ofthe data streams at the periodicity specified for the data streamlanguage program. The grouping module 360 executes the groupby block1735 to group the data streams identified by the find block 1710 by thedatacenter values into a set of data streams, each data stream of theset corresponding to a datacenter value. The window module 380 executesthe window block 1740 to identify data points corresponding to a onehour moving window for each data stream input to the window block 1740.The computation module 330 executes the stats block 1745 to determinethe a one hour moving average value for the one hour moving windowscorresponding to each data stream output by the window block 1740. Thecustomized block module 390 processes customized macros defined by usersby combining built-in blocks of the data stream language. Thecomputation module 330 scales the output of the stats block 1745 by afactor of 150% by executing the scale block 1750. The scaled output ofthe scale block 1750 is provided as input to the high port of thethreshold block 1760.

Accordingly, the threshold module 340 compares a set of result datastreams representing the mean of data streams from each datacenter witha one hour moving average of the data of data streams from each datacenter scaled by 150%. If the data of a result data stream correspondingto a datacenter received by the in port exceeds the scaled movingaverage value of the data streams for the same data center received atthe high port of the threshold block 1760, the threshold module 340generates an event. Accordingly, FIG. 17 shows an example of a datastream language program illustrating generation of a dynamicallychanging set of data streams received as input and a dynamicallychanging set of data streams provided as threshold values forcomparison.

FIG. 18 shows a flowchart illustrating the process of executing a datastream language program including a threshold block, according to anembodiment. The threshold module 340 identifies 1810 a threshold blockof a data stream language program being executed. The threshold module340 identifies 1820 various components and parameters describing thethreshold block including the input ports, the low/high threshold ports,the size of a threshold window is specified, and a fraction valueassociated with the threshold window if specified. In some embodiments,the low and/or high thresholds may be constant values in which case,either a constant value is specified as input to the low/high thresholdports or the low/high threshold values are specified as parameters ofthe threshold block (without specifying any low/high threshold ports.)

The data stream language processor 200 executes the portion of the datastream language program providing input to the input port and theportion of the data stream language program providing inputs to thelow/high threshold ports. This execution is repeated based on theperiodicity specified for the job corresponding to the data streamlanguage program. The threshold module 340 performs the comparison ofdata received in the input ports against data received in the low/highthreshold ports for each time interval based on the periodicity of thedata stream language program. If the portion of the data stream languageprogram providing input to the input port (or the low or high thresholdport) includes a groupby block, the input port of the threshold blockreceives a group of data streams. The number of data streams at eachport depends on the distinct values of the metadata attribute (or a setof metadata attributes) specified in the corresponding groupby block(provided there is at least one data stream in the input of the groupbyblock having that distinct value of the metadata attribute).

In an embodiment, the data stream language processor 200 analyzes theblocks providing data at the input port and low/high threshold ports toidentify the last groupby block that occurs before data is input to thethreshold block. The threshold module 340 uses the last groupby block toidentify the data streams received at each port, for example, to matchdata streams from the input port against data streams from the lowand/or high threshold ports and to identify data streams in events if anevent is generated based on a data stream. The threshold module 340determines that two data streams received at two different ports of thethreshold block are matching if they have the same distinct value of themetadata attribute used by the groupby block. For example, if thegroupby block used by the data stream language program for generatingdata streams provided as input to two ports of the threshold block groupdata streams based on datacenter attribute, the data streams obtained byaggregating data of a particular datacenter (say datacenter east, ordatacenter west) are determined to match.

The threshold module 340 performs the following computation for eachdata stream received at each port (i.e., the input port, the low port,and the high port). The threshold module 340 compares 1850 the datavalues received at the input port with the data values received at thelow threshold port and/or the high threshold port. The threshold module340 generates an event if the data value received at the input porteither exceeds the data value received at the high threshold port or isbelow the data value received at the low threshold port. The generatedevent includes information identifying the data streams received at theinput port based on the value of the metadata attribute corresponding tothe data stream.

In an embodiment, the data port of the threshold block receives a firstplurality of data streams generated as a result of grouping an input setof data streams based on a group by command that groups the input set ofdata streams based on a first set of metadata attributes (for example,region and data_center). The threshold port of the threshold blockreceives a second plurality of data streams generated as a result ofgrouping an input set of data streams based on a group by command thatgroups the input set of data streams based on a second set of metadataattributes. The second set of metadata attributes may be same as thefirst set of metadata attributes. Alternatively, the second set ofmetadata attributes may be different from the first set of metadataattributes. In particular, the second set of metadata attributes may bea subset of the first set of metadata attributes. For example, if thefirst set of metadata attributes includes region and data_center, thesecond set of metadata attributes includes only regions. As anotherexample, the first set of metadata attributes includes region,data_center, machine_id the second set of metadata attributes includesonly region and data_center. Accordingly, the threshold input receivesfewer data streams than the data input of the threshold block. As aresult, a plurality of data streams received at the data port may becompared with the same data stream received at the threshold port. Inthe above example, the data port receives a data stream for eachdistinct combination of values of region, data_center, machine_id andthe threshold port receives a data stream for each distinct combinationof values of region, data_center. Accordingly, all data streamscorresponding to a region and data_center received at the data port arecompared against the same data stream received at the threshold portirrespective of the machine_id value associated with the data streamreceived at the data port.

If the threshold block specifies a threshold window, the thresholdmodule 340 compares all data points at the input port received duringthe last window of the specified threshold window size against the datavalue received at the low and/or high threshold port. If all the datavalues occurring during the identified window lie outside the specifiedboundaries based on the threshold (i.e., are either greater than thehigh threshold or below the low threshold), the threshold blockgenerates an event.

If the threshold block specifies a fraction parameter in addition to thethreshold window size, the threshold module 340 compares the data pointsreceived at the input port during the last window of the specifiedthreshold window size against the data value received at the low and/orhigh threshold port. The threshold module 340 generates an event if morethan the specified fraction of data points from the identified windoware outside the bounds specified by the threshold block. For example, ifthe fraction value is 0.75 (i.e., 75%), the threshold module 340generates an event if more than 75% of data points from the identifiedwindow are outside the bounds specified by the threshold block. In anembodiment, the threshold module 340 generates an event if data pointsoccurring during more than the specified fraction of the identifiedwindow are outside the bounds specified by the threshold block. Forexample, if the fraction value is 0.75 (i.e., 75%), the threshold module340 generates an event if data points occurring during more than 75% ofthe identified window are outside the bounds specified by the thresholdblock.

Customized Blocks for Data Stream Language Programs

A customized block can be specified by a user by combining existingbuilt-in blocks of the data stream language. A customized block is alsoreferred to as a macro block or a customized macro block. The ability todefine customized macro blocks makes the data stream languageextensible. A customized block can be included in a data stream languageprogram similar to the built-in blocks. A customized block can use othercustomized macro blocks allowing arbitrary nesting of customized macroblocks. A user can specify arbitrary abstractions using customizedblocks. A customized block is executed at the periodicity specified forthe job executing the data stream language program including thecustomized macro block. The customized block module 390 determines theinput values for each input port of the customized macro block for eachtime interval.

The customized block module 390 executes the instructions of thecustomized macro block and generates data values for each output port.The output values from the output port may be provided to subsequentblocks. If an input to the customized block comprises blocks including agroupby block, the input port may receive a plurality of data streams asinput. The customized block module 390 executes the instructions of thecustomized block module 390 for each data point of each data streamreceived at the input. The number of data streams may be dynamicallychanging based on changes in the overall set of data streams received bythe data stream language program including the customized macro block. Acustomized macro block may be associated with one or more parametersthat are used in the instructions of the customized block. Theinstructions of the customized macro block use parameter values.However, when the customized macro block is specified in a data streamlanguage program, specific values for each parameter are provided.Accordingly, the customized block module 390 substitutes the parameternames for the parameter values while executing the instructions of thecustomized macro block.

FIG. 19 shows an example of a data stream language program illustratinguse of a customized block for generating a result data stream based on auser defined function applied to inputs comprising groups of datastreams, according to an embodiment. The example customized macro block1960 combines data of two input data streams to generate a functionbased on the input data values. The combine block 1960 has two inputports hits and misses and one output port out. The input to each inputport is generated by a portion of the data stream language program.

For example, the input to the input port hits is generated as output ofthe stats block 1925 and the input of the input port misses is generatedas output of the starts block 1945. The find module 310 executes thefind block 1900 to find all data streams received by the instrumentationanalysis system 100 that have the metric values cacheHits. For example,the find module 310 may execute the find block 1900 to find all datastreams received from development systems 120 that provide values ofcache hits. The fetch module 320 executes the fetch block 1915 to fetchthe data of the data streams identified by the find block 1900. Thegrouping module executes the groupby block 1920 to group the datastreams by datacenter attribute. The computation module 330 executes thestats block 1925 to generate the mean of data from all data streams foreach distinct datacenter and provides the data as input to the hits portof the combine block 1960.

Similarly, the find module 310 executes the find block 1910 to find alldata streams received by the instrumentation analysis system 100 thathave the metric values cacheMisses. For example, the find module 310 mayexecute the find block 1910 to find all data streams received fromdevelopment systems 120 that provide values of cache misses. The fetchmodule 320 executes the fetch block 1930 to fetch the data of the datastreams identified by the find block 1900. The grouping module executesthe groupby block 1935 to group the data streams by the datacenterattribute. The computation module 330 executes the stats block 1945 togenerate the mean of data from all data streams for each distinctdatacenter and provides the data as input to the hits port of thecombine block 1960.

The customized block module 390 executes the set of instructions 1910specified for the combine block. Accordingly, for each time interval,the customized block module 390 determines the value of H/(H+M) if Hrepresents the data value received at the hits input port and Mrepresents the value of misses received at the misses port. Thecustomized block module 390 provides the value of the above expressionto the output port. The data stream language processor 200 provides thedata values from the output port to the input port of a subsequentblock, if any.

FIG. 20 shows a flowchart illustrating the process of executing a datastream language program with a customized block, according to anembodiment. The data stream language processor identifies 2010 acustomized blocks of data stream language program. The customized blockmodule 390 identifies 2020 the input ports and the output ports of thecustomized block. If the customized block specifies parameter values,the customized block module 390 receives values to be substitutes forthe parameters and substitutes them in the instructions specified by thecustomized block.

The customized block module 390 repeats the following steps for eachtime interval. The customized block module 390 determines the input datavalue for each input port. If the portion of the data stream languageprogram generating input for an input port includes a groupby block, theinput to the port may comprise multiple data values corresponding toeach data stream generated by the groupby block.

The customized block module 390 executes the instructions of thecustomized block for each data value. If there are multiple data streamsinput at each port, the customized block module 390 identifies matchingdata streams by comparing the values of the metadata attribute of thegroupby blocks for each input port. The customized block module 390executes 2040 the instructions for each data stream that is input to theinput ports. If an input port has a constant input value and anotherinput port has a plurality of data streams, the customized block module390 applies the constant value to each data stream of the other inputport.

The customized block module 390 provides the value of the result ofexecution of the instructions of the customized block to the outputports as specified in the instructions of the customized block. The datastream language processor 200 provides the values at the output ports tothe blocks of the data stream language program connected to the outputports. A customized block may output multiple data streams at an outputport. For example, the input ports of the customized block may eachreceives multiple data streams and the customized block may perform aparticular computation on tuples comprising values from matching datastreams received at each input port.

The instructions of a customized data block may include other customizeddata blocks. Accordingly, the above process illustrated in FIG. 20 isexecuted for each customized block.

User Interface for Generating Reports Using Data Stream LanguagePrograms

In some embodiments, the instrumentation analysis system 100 provides auser interface that generates data stream language programs for the enduser interested in viewing the reports based on data streams. The useris provided with a user friendly user interface that hides thecomplexity of the data stream language. The user interface provided bythe instrumentation analysis system shows various widgets that allowusers to take actions such as select the metrics for generating reports,performing rollups, grouping data streams and so on.

FIG. 21 shows a screenshot of a user interface displaying result ofexecution of a data stream language program that shows data streamsreceived by the instrumentation analysis system, according to anembodiment. The screenshot shows several charts 2120 displaying datastreams representing metric 2120 service.cache.hits. The metricrepresents cache hit values received from instrumented softwareexecuting on development systems 120. The values are rolled up to a timeinterval of 1 second. Accordingly, the cache hits values received ineach time interval of one second are added together. There can be alarge number of services reporting the metric service.cache.hits andaccordingly a large number of charts 2120 is displayed. FIG. 21 showsvarious widgets that allow a user to take actions, for example, selectthe metric that is reported by the user interface, perform rollups.

FIG. 22 shows a screenshot of a user interface displaying result ofexecution of a data stream language program showing 1 minute average ofdata of data streams received by the instrumentation analysis system,according to an embodiment. FIG. 22 shows a widget that allows a user tospecify certain computations to be performed on the data streams.Specifically, FIG. 22 shows a widget 2220 that computes a one minutemean for each data stream. As a result the charts 2210 are smoother thanthe charts shown in FIG. 21. However the number of charts 2210 shown inFIG. 22 is same as the number of charts 2210 shown in FIG. 21.

Large enterprises may have a very large number of development systems120. Each development system may execute multiple services, each servicereporting the metrics. As a result, the number of charts displayed inFIGS. 21 and 22 can be very large. A user can gain better insight intothe data reported by data streams by grouping the data streams as shownin FIG. 23.

FIG. 23 shows a screenshot of a user interface displaying result ofexecution of a data stream language program showing sum of data streamsgrouped by data center, according to an embodiment. FIG. 23 shows widget2320 that allows specification of attribute by which the data streamsare grouped and the aggregation operation performed for each group. Asshown in FIG. 23, the user has requested grouping by data center andperforming the sum operation for each group. Assuming there are only twodata centers, the number of charts is reduced to two. Each chart 2310shows the sum of data values of data streams received from a particulardata center.

FIG. 24 shows a screenshot of a user interface displaying result ofexecution of a data stream language program including a customized macroblock that determines ratio of cache hit rate and sum of cache hit rateand miss rate, for data streams grouped by datacenters, according to anembodiment. As shown in FIG. 24, a user refers to data streams reportingmetric service.cache.hit using the widget 2430 as A. The user furtherrefers to data streams reporting the metric service.cache.miss using thewidget 2440 as B. The user defines the computation A/(A+B) as the ratioof the cache hit with respect to the sum of cache hits and cache misses.The user further specifies using widget 2450 that the value A/(A+B)computed should be scaled by a multiple of 100. This computation isperformed for each group of data streams based on datacenter.Accordingly, a chart 2410 is generated for each data center reportingreal time values of cache hit ratio for all data streams received fromthe data center.

Alternative Embodiments

It is to be understood that the Figures and descriptions of the presentinvention have been simplified to illustrate elements that are relevantfor a clear understanding of the invention, while eliminating, for thepurpose of clarity, many other elements found in a typical system. Thoseof ordinary skill in the art may recognize that other elements and/orsteps are desirable and/or required in implementing the presentinvention. However, because such elements and steps are well known inthe art, and because they do not facilitate a better understanding ofthe present invention, a discussion of such elements and steps is notprovided herein. The disclosure herein is directed to all suchvariations and modifications to such elements and methods known to thoseskilled in the art.

Some portions of above description describe the embodiments in terms ofalgorithms and symbolic representations of operations on information.These algorithmic descriptions and representations are commonly used bythose skilled in the data processing arts to convey the substance oftheir work effectively to others skilled in the art. These operations,while described functionally, computationally, or logically, areunderstood to be implemented by computer programs or equivalentelectrical circuits, microcode, or the like. Furthermore, it has alsoproven convenient at times, to refer to these arrangements of operationsas modules, without loss of generality. The described operations andtheir associated modules may be embodied in software, firmware,hardware, or any combinations thereof.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. It should be understood thatthese terms are not intended as synonyms for each other. For example,some embodiments may be described using the term “connected” to indicatethat two or more elements are in direct physical or electrical contactwith each other. In another example, some embodiments may be describedusing the term “coupled” to indicate that two or more elements are indirect physical or electrical contact. The term “coupled,” however, mayalso mean that two or more elements are not in direct contact with eachother, but yet still co-operate or interact with each other. Theembodiments are not limited in this context.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the invention. Thisdescription should be read to include one or at least one and thesingular also includes the plural unless it is obvious that it is meantotherwise. Upon reading this disclosure, those of skill in the art willappreciate still additional alternative structural and functionaldesigns for a system and a process for generating reports based oninstrumented software through the disclosed principles herein. Thus,while particular embodiments and applications have been illustrated anddescribed, it is to be understood that the disclosed embodiments are notlimited to the precise construction and components disclosed herein.Various modifications, changes and variations, which will be apparent tothose skilled in the art, may be made in the arrangement, operation anddetails of the method and apparatus disclosed herein without departingfrom the spirit and scope defined in the appended claims.

We claim:
 1. A method for detecting anomalies in data streams, themethod comprising: receiving instructions of a data stream languageprogram, comprising: a threshold block configured to receive a firstinput comprising a plurality of data streams comprising data values anda second input comprising a plurality of data streams comprisingthreshold values; a first set of instructions for generating the firstinput of the threshold block; and a second set of instructions forgenerating the second input of the threshold block; receiving an inputset of data streams; and executing the first set of instructions toaggregate data of the input set of data streams to generate a firstplurality of data streams comprising data values provided as the firstinput of the threshold block, wherein executing the first set ofinstructions comprises grouping the input set of data streams based on afirst set of metadata attributes and wherein the number of data streamsin the first plurality of data streams depends on the number of distinctvalues of the first set of metadata attributes; executing the second setof instructions to aggregate data of the input set of data streams togenerate a second plurality of data streams comprising threshold valuesprovided as the second input of the threshold block, wherein executingthe second set of instructions comprises grouping the input set of datastreams based on a second set of metadata attributes and wherein thenumber of data streams in the second plurality of data streams dependson the number of distinct values of the second set of metadataattributes; matching data streams from the first plurality with datastreams from the second plurality to identify corresponding datastreams; for each data stream of the first plurality: comparing a datavalue of the data stream against a threshold value from a correspondingdata stream from the second plurality; and determining whether togenerate an event based on the result of comparison of the data valueand the threshold value.
 2. The method of claim 1, wherein determiningwhether to generate an event comprises, generating an event if thecomparison indicates that the data of a data stream of the input groupexceeds a threshold value of the corresponding data stream from thesecond plurality of data streams.
 3. The method of claim 1, wherein eachof the first plurality of data streams is associated with a first set ofvalues of the first set of metadata attributes each of the secondplurality of data streams is associated with a second set of values ofthe second set of metadata attributes.
 4. The method of claim 3, whereinmatching data streams from the first plurality with data streams fromthe second plurality comprises comparing values from the first set ofvalues with values from the second set of values.
 5. The method of claim1, wherein the threshold values of the data streams of the secondplurality of data streams are high threshold values and an event isgenerated if the data of a data stream of the first plurality of datastreams exceeds a threshold value from a corresponding data stream fromthe second plurality.
 6. The method of claim 1, wherein the thresholdvalues of the data streams of the second plurality of data streams arelow threshold values and an event is generated if the data of a datastream of the first plurality of data streams is determined to be belowa threshold value from a corresponding data stream from the secondplurality.
 7. The method of claim 1, wherein the second set of metadataattributes is a subset of the first set of metadata attributes.
 8. Themethod of claim 1, wherein the second set of instructions comprises awindow instruction to determine a moving aggregate value based on amoving window of a predetermined size.
 9. The method of claim 1, whereinthe threshold block is associated with a threshold window size, whereinthe event is generated if a data value of a data stream from the firstplurality of data streams exceeds the data value of the matching datastream from the second plurality of data streams for all data pointsoccurring in a window of the threshold window size.
 10. The method ofclaim 9, wherein the threshold block is further associated with afraction parameter, wherein the event is generated if the data values ofa data stream from the first plurality of data streams are greater thanthe data values of the matching data stream from the second pluralityfor more than the fraction of data points occurring in a window of thethreshold window size.
 11. The method of claim 9, wherein the thresholdblock is further associated with a fraction parameter, wherein the eventis generated if the data values of a data stream from the input groupare greater than the data values of the matching data stream from thesecond plurality for more than the fraction of time duration of a windowof the threshold window size.
 12. The method of claim 1, furthercomprising: determining whether each data stream of the first pluralityhas a matching data stream in the second plurality; and sending an errormessage if there is at least a data stream in the first pluralitywithout a matching data stream in the second plurality.
 13. The methodof claim 1, wherein events are generated for a first subset of the firstplurality of data streams but are not generated for a second subset ofthe first plurality of data streams.
 14. The method of claim 1, whereinthe data streams in the first plurality and the second plurality changedynamically as a result of changes in the set of data streams.
 15. Themethod of claim 1, wherein the data streams in first plurality and thesecond plurality change dynamically as a result of changes in the valuesof metadata attributes describing data streams.
 16. The method of claim1, wherein the data streams of the input set of data streams arereceived from one or more of instrumented software executing on externalsystems or result of execution of a data stream language program.
 17. Acomputer-readable non-transitory storage medium storing instructionsfor: receiving instructions of a data stream language program,comprising: a threshold block configured to receive a first inputcomprising a plurality of data streams comprising data values and asecond input comprising a plurality of data streams comprising thresholdvalues; a first set of instructions for generating the first input ofthe threshold block; and a second set of instructions for generating thesecond input of the threshold block; receiving an input set of datastreams; and executing the first set of instructions to aggregate dataof the input set of data streams to generate a first plurality of datastreams comprising data values provided as the first input of thethreshold block, wherein executing the first set of instructionscomprises grouping the input set of data streams based on a first set ofmetadata attributes and wherein the number of data streams in the firstplurality of data streams depends on the number of distinct values ofthe first set of metadata attributes; executing the second set ofinstructions to aggregate data of the input set of data streams togenerate a second plurality of data streams comprising threshold valuesprovided as the second input of the threshold block, wherein executingthe second set of instructions comprises grouping the input set of datastreams based on a second set of metadata attributes and wherein thenumber of data streams in the second plurality of data streams dependson the number of distinct values of the second set of metadataattributes; matching data streams from the first plurality with datastreams from the second plurality to identify corresponding datastreams; and for each data stream of the first plurality: comparing adata value of the data stream against a threshold value from acorresponding data stream from the second plurality; and determiningwhether to generate an event based on the result of comparison of thedata value and the threshold value.
 18. The computer-readablenon-transitory storage medium of claim 17, wherein determining whetherto generate an event comprises, generating an event if the comparisonindicates that the data of a data stream of the input group exceeds athreshold value of the corresponding data stream from the secondplurality of data streams.
 19. The computer-readable non-transitorystorage medium of claim 17, wherein each of the first plurality of datastreams is associated with a first set of values of the first set ofmetadata attributes each of the second plurality of data streams isassociated with a second set of values of the second set of metadataattributes.
 20. The computer-readable non-transitory storage medium ofclaim 19, wherein matching data streams from the first plurality withdata streams from the second plurality comprises comparing values fromthe first set of values with values from the second set of values. 21.The computer-readable non-transitory storage medium of claim 17, whereinthe threshold values of the data streams of the second plurality of datastreams are high threshold values and an event is generated if the dataof a data stream of the first plurality of data streams exceeds athreshold value from a corresponding data stream from the secondplurality.
 22. The computer-readable non-transitory storage medium ofclaim 17, wherein the threshold values of the data streams of the secondplurality of data streams are low threshold values and an event isgenerated if the data of a data stream of the first plurality of datastreams is determined to be below a threshold value from a correspondingdata stream from the second plurality.
 23. The computer-readablenon-transitory storage medium of claim 17, wherein the storedinstructions are further for: determining whether each data stream ofthe first plurality has a matching data stream in the second plurality;and sending an error message if there is at least a data stream in thefirst plurality without a matching data stream in the second plurality.24. A computer system comprising: a computer processor; and acomputer-readable non-transitory storage medium storing instructionsfor: receiving instructions of a data stream language program,comprising: a threshold block configured to receive a first inputcomprising a plurality of data streams comprising data values and asecond input comprising a plurality of data streams comprising thresholdvalues; a first set of instructions for generating the first input ofthe threshold block; and a second set of instructions for generating thesecond input of the threshold block; receiving an input set of datastreams; and executing the first set of instructions to aggregate dataof the input set of data streams to generate a first plurality of datastreams comprising data values provided as the first input of thethreshold block, wherein executing the first set of instructionscomprises grouping the input set of data streams based on a first set ofmetadata attributes and wherein the number of data streams in the firstplurality of data streams depends on the number of distinct values ofthe first set of metadata attributes; executing the second set ofinstructions to aggregate data of the input set of data streams togenerate a second plurality of data streams comprising threshold valuesprovided as the second input of the threshold block, wherein executingthe second set of instructions comprises grouping the input set of datastreams based on a second set of metadata attributes and wherein thenumber of data streams in the second plurality of data streams dependson the number of distinct values of the second set of metadataattributes; matching data streams from the first plurality with datastreams from the second plurality to identify corresponding datastreams; and for each data stream of the first plurality: comparing adata value of the data stream against a threshold value from acorresponding data stream from the second plurality; and determiningwhether to generate an event based on the result of comparison of thedata value and the threshold value.
 25. The computer system of claim 24,wherein determining whether to generate an event comprises, generatingan event if the comparison indicates that the data of a data stream ofthe input group exceeds a threshold value of the corresponding datastream from the second plurality of data streams.
 26. The computersystem of claim 24, wherein each of the first plurality of data streamsis associated with a first set of values of the first set of metadataattributes each of the second plurality of data streams is associatedwith a second set of values of the second set of metadata attributes.27. The computer system of claim 26, wherein matching data streams fromthe first plurality with data streams from the second pluralitycomprises comparing values from the first set of values with values fromthe second set of values.
 28. The computer system of claim 24, whereinthe threshold values of the data streams of the second plurality of datastreams are high threshold values and an event is generated if the dataof a data stream of the first plurality of data streams exceeds athreshold value from a corresponding data stream from the secondplurality.
 29. The computer system of claim 24, wherein the thresholdvalues of the data streams of the second plurality of data streams arelow threshold values and an event is generated if the data of a datastream of the first plurality of data streams is determined to be belowa threshold value from a corresponding data stream from the secondplurality.
 30. The computer system of claim 24, wherein the storedinstructions are further for: determining whether each data stream ofthe first plurality has a matching data stream in the second plurality;and sending an error message if there is at least a data stream in thefirst plurality without a matching data stream in the second plurality.